Re: execute commands as su on remote server
On Tuesday, 18 August 2015 08:27:33 UTC+5:30, hariramm...@gmail.com wrote: > execute commands as su on remote server > > Postby hariram » Mon Aug 17, 2015 4:02 am > Needed: > I need to execute commands after doing su to other user on remote server(not > sudo which doesn't require password) how i can achieve this using python? > I googled and came to know that its not possible, so just for confirmation > asking again, is it possible ? > > Already Tried: > Tried paramiko that's too not working. Hey Chris, Thanks for that, requirement is not to use sudo, we have to use only su. anyway i have tried with subprocess too with su which is not working as it needs tty...even paramiko also SSHClient... -- https://mail.python.org/mailman/listinfo/python-list
Re: execute commands as su on remote server
On Tuesday, 18 August 2015 08:27:33 UTC+5:30, hariramm...@gmail.com wrote: > execute commands as su on remote server > > Postby hariram » Mon Aug 17, 2015 4:02 am > Needed: > I need to execute commands after doing su to other user on remote server(not > sudo which doesn't require password) how i can achieve this using python? > I googled and came to know that its not possible, so just for confirmation > asking again, is it possible ? > > Already Tried: > Tried paramiko that's too not working. Hey Julio, i have seen that in forums speaking about fabfile... but i believe that it also supports only sudo not su...sudo rights we dont have in our env... even fabfile implementation is up to python 2.7 only and we are using python 3.4 Tried creating channel and invoking shell and spawning it delay, that too didnt worked, just checking that any possible way is there to control the tty.. -- https://mail.python.org/mailman/listinfo/python-list
Re: Logging to a file from a C-extension
In a message of Tue, 18 Aug 2015 08:07:51 -0500, Al Pfalzgraf writes:
> If a logging file is opened at the level of a Python application, how would
> the log file name be communicated to a C-extension so that logging from the
> extension would be sent to the same log file?
To convert a file to an integer file descriptor, use PyFile_FromFd()
PyObject *fobj; /* File object (already obtained somehow) */
int fd = PyObject_AsFileDescriptor(fobj);
if (fd < 0) {
return NULL;
}
This works for files, sockets, anything that produces a file descriptor.
Remember to flush your file on the Python side before you hand it to your
C extension, or your data will arrive scrambled, or you can lose some.
If you need to go the other way:
int fd; /* Existing file descriptor (already open) */
PyObject *fobj = PyFile_FromFd(fd, "filename","r",-1,NULL,NULL,NULL,1);
--
https://mail.python.org/mailman/listinfo/python-list
Re: execute commands as su on remote server
I haven't tried this but fabric looks encouraging: >From >http://docs.fabfile.org/en/latest/api/core/operations.html#fabric.operations.run fabric.operations.run(*args, **kwargs) Run a shell command on a remote host. ... Any text entered in your local terminal will be forwarded to the remote program as it runs, thus allowing you to interact with password or other prompts naturally ... You may pass pty=False to forego creation of a pseudo-terminal on the remote end in case the presence of one causes problems for the command in question. Laura -- https://mail.python.org/mailman/listinfo/python-list
Re: execute commands as su on remote server
In a message of Wed, 19 Aug 2015 10:44:53 +0200, Laura Creighton writes: >I haven't tried this but fabric looks encouraging: > >>From >>http://docs.fabfile.org/en/latest/api/core/operations.html#fabric.operations.run > > fabric.operations.run(*args, **kwargs) > >Run a shell command on a remote host. > >... > >Any text entered in your local terminal will be forwarded to the >remote program as it runs, thus allowing you to interact with >password or other prompts naturally > >... > >You may pass pty=False to forego creation of a pseudo-terminal on >the remote end in case the presence of one causes problems for the >command in question. > >Laura >-- >https://mail.python.org/mailman/listinfo/python-list And now that I started looking, I found this: http://stackoverflow.com/questions/12641514/switch-to-different-user-using-fabric so, somebody got it to work ... Laura -- https://mail.python.org/mailman/listinfo/python-list
Permission denied error in download nltk_data...
Please help
Using this code:
import nltk
nltk.download('maxent_treebank_pos_tagger¹)
I get this error:
[nltk_data] Downloading package maxent_treebank_pos_tagger to
[nltk_data]/Users/dwightgoldwindex/nltk_data...
Traceback (most recent call last):
File "test short.py", line 18, in
nltk.download('maxent_treebank_pos_tagger')
File
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packag
es/nltk/downloader.py", line 664, in download
for msg in self.incr_download(info_or_id, download_dir, force):
File
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packag
es/nltk/downloader.py", line 549, in incr_download
for msg in self._download_package(info, download_dir, force):
File
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packag
es/nltk/downloader.py", line 604, in _download_package
os.mkdir(download_dir)
PermissionError: [Errno 13] Permission denied:
'/Users/dwightgoldwindex/nltk_data'
BIG SMILE...
Always, Dwight
www.3forliving.key.to (video playlist on YouTube)
www.couragebooks.key.to (all my books on Amazon)
--
https://mail.python.org/mailman/listinfo/python-list
ANN: eGenix mxODBC 3.3.5 - Python ODBC Database Interface
ANNOUNCING eGenix.com mxODBC Python ODBC Database Interface Version 3.3.5 mxODBC is our commercially supported Python extension providing ODBC database connectivity to Python applications on Windows, Mac OS X, Unix and BSD platforms with many advanced Python DB-API extensions and full support of stored procedures This announcement is also available on our web-site for online reading: http://www.egenix.com/company/news/eGenix-mxODBC-3.3.5-GA.html INTRODUCTION mxODBC provides an easy-to-use, high-performance, reliable and robust Python interface to ODBC compatible databases such as MS SQL Server, Oracle Database, IBM DB2, Informix and Netezza, SAP Sybase ASE and Sybase Anywhere, Teradata, MySQL, MariaDB, PostgreSQL, SAP MaxDB and many more: http://www.egenix.com/products/python/mxODBC/ The "eGenix mxODBC - Python ODBC Database Interface" product is a commercial extension to our open-source eGenix mx Base Distribution: http://www.egenix.com/products/python/mxBase/ NEWS The 3.3.5 release of our mxODBC is a patch level release of our popular Python ODBC Interface for Windows, Linux, Mac OS X and FreeBSD. It includes these enhancements and fixes: Features * Documented the use of transaction isolation levels with mxODBC in a new section of the mxODBC manual. This features has been part of mxODBC for long time, but was never documented as such. Driver Compatibility MS SQL Server * Fixed the definition of the BinaryNull singleton added in mxODBC 3.3.4 to make it pickleable and protect it against recreation. * Documented and recommended use of SET NOCOUNT ON for running multiple statements or stored procedures. This can not only resolve issues with error reporting, it also results in better performance. Bug Fixes - * Fixed a potential segfault during interpreter shutdown introduced in mxODBC 3.3.4. Found by ZeOmega while testing mxODBC with SQLAlchemy (SA) using the "mssql+mxodbc" SA engine Installation Enhancements - * Added support for bdist_wheels to mxSetup, which is used for creating distribution packages of mxODBC, to allow building wheels from the prebuilt packages, e.g. during installation via pip. For the full set of changes please check the mxODBC change log: http://www.egenix.com/products/python/mxODBC/changelog.html FEATURES mxODBC 3.3 was released on 2014-04-08. Please see the full announcement for highlights of the 3.3 release: http://www.egenix.com/company/news/eGenix-mxODBC-3.3.0-GA.html For the full set of features mxODBC has to offer, please see: http://www.egenix.com/products/python/mxODBC/#Features EDITIONS mxODBC is available in these two editions: * The Professional Edition, which gives full access to all mxODBC features. * The Product Development Edition, which allows including mxODBC in applications you develop. For a complete overview of the available editions, please see the product page: http://www.egenix.com/products/python/mxODBC/#mxODBCEditions DOWNLOADS The download archives and instructions for installing the package can be found at: http://www.egenix.com/products/python/mxODBC/ In order to use the eGenix mxODBC package you will first need to install the eGenix mx Base package: http://www.egenix.com/products/python/mxBase/ You can also simply use: pip install egenix-mxodbc and then get evaluation licenses from our website to try mxODBC: http://www.egenix.com/products/python/mxODBC/#Evaluation UPGRADING Users are encouraged to upgrade to this latest mxODBC release to benefit from the new features and updated ODBC driver support. We have taken special care not to introduce backwards incompatible changes, making the upgrade experience as smooth as possible. Customers who have purchased mxODBC 3.3 licenses can continue to use their licenses with this patch level release. For upgrade purchases, we will give out 20% discount coupons going from mxODBC 2.x to 3.3 and 50% coupons for upgrades from mxODBC 3.x to 3.3. Please contact the eGenix.com Sales Team with your existing license serials for details for an upgrade discount coupon. If you want to try the new release before purchase, you can request 30-day evaluation licenses by visiting our web-site http://www.egenix.co
Why the different parameter signatures to indicate blocking/timeouts.
I have been looking through the threading module and I am a bit annoyed at the different signatures for indicating whether some calls should block or not and the timeout. On the one hand we have the acquire on locks with the following signature: def accquire(blocking=True, timeout=-1) On the other hand we have wait and join methods with this signature: def join(timeout=None). As far as I understand timeout=None is equivallent to: blocking=True, timeout=-1 timeout=NR is equivallent to: blocking=True, timeout=NR. timeout=0is equivallent to: blocking=False, timeout=-1 which may be equivallent to : blocking=True, timeout=0 Can someone explain why these different signatures, or should I consider this a relict from the past? -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
Can I download XML data from the web and save, in as CSV or TXT delimitation?
I'm trying to get R to download the data from here:
http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml
# install and load the necessary package
install.packages("XML")
library(XML)
# Save the URL of the xml file in a variable
xml.url <-
"http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml";
# Use the xmlTreePares-function to parse xml file directly from the web
xmlfile <- xmlTreeParse(xml.url)
# the xml file is now saved as an object you can easily work with in R:
class(xmlfile)
# Use the xmlRoot-function to access the top node
xmltop = xmlRoot(xmlfile)
# have a look at the XML-code of the first subnodes:
print(xmltop)[1:3]
Everything seems fine up to that point. The next line seems to NOT parse the
data as I thought it would.
# To extract the XML-values from the document, use xmlSApply:
datacat <- xmlSApply(xmltop, function(x) xmlSApply(x, xmlValue))
I did some research on this, and it seemed to work in other examples of xml
data. I guess this data set is different...or I just don't understand this well
enough to know what's really going on...
Basically, I want to get this:
xmltop
Into a data table. How can I do that?
Thanks.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Permission denied error in download nltk_data...
On Wed, 19 Aug 2015 04:57 pm, Dwight GoldWinde wrote:
> Please help
>
> Using this code:
>
> import nltk
> nltk.download('maxent_treebank_pos_tagger¹)
>
>
> I get this error:
[...]
>os.mkdir(download_dir)
>
> PermissionError: [Errno 13] Permission denied:
> '/Users/dwightgoldwindex/nltk_data'
That's a permissions problem.
Firstly, when you run this code, are you logged in as the "dwightgoldwindex"
user? Note the "x" at the end of the directory name. Is that what you
expect?
Secondly, can you open a terminal and get a command line prompt? **NOT** a
Python prompt: if you see the prompt ">>>" type "quit()" and enter. You
should have a prompt $ or % or similar.
At the prompt, type:
whoami
ls -ld /Users/dwightgoldwindex/
ls -ld /Users/dwightgoldwindex/nltk_data
and hit Enter after each line. What results do you get?
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Can I download XML data from the web and save, in as CSV or TXT delimitation?
In a message of Wed, 19 Aug 2015 04:57:44 -0700, ryguy7272 writes:
>I'm trying to get R to download the data from here:
>
>http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml
>
>
># install and load the necessary package
>install.packages("XML")
>library(XML)
># Save the URL of the xml file in a variable
>
>xml.url <-
>"http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml";
># Use the xmlTreePares-function to parse xml file directly from the web
>
>xmlfile <- xmlTreeParse(xml.url)
># the xml file is now saved as an object you can easily work with in R:
>class(xmlfile)
>
>
># Use the xmlRoot-function to access the top node
>xmltop = xmlRoot(xmlfile)
># have a look at the XML-code of the first subnodes:
>print(xmltop)[1:3]
>
>
>
>Everything seems fine up to that point. The next line seems to NOT parse the
>data as I thought it would.
># To extract the XML-values from the document, use xmlSApply:
>datacat <- xmlSApply(xmltop, function(x) xmlSApply(x, xmlValue))
>
>
>
>I did some research on this, and it seemed to work in other examples of xml
>data. I guess this data set is different...or I just don't understand this
>well enough to know what's really going on...
>
>Basically, I want to get this:
>
>xmltop
>
>
>Into a data table. How can I do that?
>
>Thanks.
This is a mailing list about the Python programming language, not R
xmlSApply is something R uses. The R mailing lists are here:
https://www.r-project.org/mail.html
When you talk to them, tell them exactly what you were expecting as
a result, what you got instead, and what error messages were generated.
Also let them know what verison of R you are using and what operating
system you are running on. This will make it a lot easier for them
to help you.
Good luck,
Laura Creighton
--
https://mail.python.org/mailman/listinfo/python-list
Re: Can I download XML data from the web and save, in as CSV or TXT delimitation?
On Wednesday, August 19, 2015 at 8:21:50 AM UTC-4, Laura Creighton wrote:
> In a message of Wed, 19 Aug 2015 04:57:44 -0700, ryguy7272 writes:
> >I'm trying to get R to download the data from here:
> >
> >http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml
> >
> >
> ># install and load the necessary package
> >install.packages("XML")
> >library(XML)
> ># Save the URL of the xml file in a variable
> >
> >xml.url <-
> >"http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml";
> ># Use the xmlTreePares-function to parse xml file directly from the web
> >
> >xmlfile <- xmlTreeParse(xml.url)
> ># the xml file is now saved as an object you can easily work with in R:
> >class(xmlfile)
> >
> >
> ># Use the xmlRoot-function to access the top node
> >xmltop = xmlRoot(xmlfile)
> ># have a look at the XML-code of the first subnodes:
> >print(xmltop)[1:3]
> >
> >
> >
> >Everything seems fine up to that point. The next line seems to NOT parse
> >the data as I thought it would.
> ># To extract the XML-values from the document, use xmlSApply:
> >datacat <- xmlSApply(xmltop, function(x) xmlSApply(x, xmlValue))
> >
> >
> >
> >I did some research on this, and it seemed to work in other examples of xml
> >data. I guess this data set is different...or I just don't understand this
> >well enough to know what's really going on...
> >
> >Basically, I want to get this:
> >
> >xmltop
> >
> >
> >Into a data table. How can I do that?
> >
> >Thanks.
>
> This is a mailing list about the Python programming language, not R
> xmlSApply is something R uses. The R mailing lists are here:
> https://www.r-project.org/mail.html
>
> When you talk to them, tell them exactly what you were expecting as
> a result, what you got instead, and what error messages were generated.
> Also let them know what verison of R you are using and what operating
> system you are running on. This will make it a lot easier for them
> to help you.
>
> Good luck,
>
> Laura Creighton
Well, yes, I was originally trying to do it it R, but I couldn't get it
working, so I thought I'd try to do it in Python. That was a sample R script.
Can I do essentially the same thing in Python? Can I read the XML from the web?
http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml
Parse it, or clean it, or whatever, and save it as a CSV or TXT? Is that
possible?
Thanks.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Can I download XML data from the web and save, in as CSV or TXT delimitation?
> > > Well, yes, I was originally trying to do it it R, but I couldn't get it > working, so I thought I'd try to do it in Python. That was a sample R > script. Can I do essentially the same thing in Python? Can I read the XML > from the web? > http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml > > Parse it, or clean it, or whatever, and save it as a CSV or TXT? Is that > possible? > > Thanks. Of course you can 1. read a file from a website using python urllib modules or requests module (3rd party but simpler) 2. There are several xml parsers. I haven't experienced them, so google is your friend 3. python has csv reader and writer modules -- Joel Goldstick http://joelgoldstick.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Can I download XML data from the web and save, in as CSV or TXT delimitation?
In a message of Wed, 19 Aug 2015 06:32:46 -0700, ryguy7272 writes: >Well, yes, I was originally trying to do it it R, but I couldn't get >it working, so I thought I'd try to do it in Python. That was a >sample R script. Can I do essentially the same thing in Python? Can >I read the XML from the web? >http://www.usda.gov/oce/commodity/wasde/report_format/latest-July-2015-New-Format.xml >Parse it, or clean it, or whatever, and save it as a CSV or TXT? Is >that possible? Thanks. -- >https://mail.python.org/mailman/listinfo/python-list The question is, is this a reasonable thing to do. If your xml file is flat, and simple, then moving to csv is easy, and a fine idea. Python has tons of utilities for dealing with xml files. For instance https://pypi.python.org/pypi/xmlutils download that and conversion is a simple as running the command line command xml2csv --input "samples/fruits.xml" --output "samples/fruits.csv" --tag "item" But XML is designed for the representation of complex data -- if your xmlfile has a lot of structure then a CSV file doesn't have enough structure in itself to do a good job of holding the data. xml2json -- you will get one of those too -- might give you a file that you would find more suitable to use. But there is a very good chance that you will have to analyse the structure of your data, and parse it, and decide what to keep and what you don't care about. Python is very, very good at doing this but you will have to learn how to program in Python -- at least a little -- to do this. Have you programmed in Python before? In another language? Laura Creighton -- https://mail.python.org/mailman/listinfo/python-list
Re: regex recursive matching (regex 2015.07.19)
MRAB writes: > On 2015-08-18 22:55, Ben Bacarisse wrote: >> Neal Becker writes: >> >>> Trying regex 2015.07.19 >>> >>> I'd like to match recursive parenthesized expressions, with groups such that >>> '(a(b)c)' >>> >>> would give >>> group(0) -> '(a(b)c)' >>> group(1) -> '(b)' >>> >>> but that's not what I get >>> >>> import regex >>> >>> #r = r'\((?>[^()]|(?R))*\)' >>> r = r'\(([^()]|(?R))*\)' >>> #r = r'\((?:[^()]|(?R))*\)' >>> m = regex.match (r, '(a(b)c)') >> >> The (?R) syntax is Perl -- it's no implemented in Python. Python and >> Perl regexs are very similar in syntax (for very good reasons) but >> neither (?R) nor the numbered or named versions of it are in Python. >> > He's using the regex module from PyPI: Ah, right. Then he might find r = r'(\((?:[^()]|(?R))*\))' more suitable when combined with captures() rather than groups(): regex.match(r, '(a(b)c)').captures(1) ['(b)', '(a(b)c)'] -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: Regular expression and substitution, unexpected duplication
MRAB wrote:
> On 2015-08-18 22:42, Laurent Pointal wrote:
>> Hello,
>> ellipfind_re = re.compile(r"((?=\.\.\.)|…)", re.IGNORECASE|re.VERBOSE)
>> ellipfind_re.sub(' ... ',
>> "C'est un essai... avec différents caractères… pour voir.")
> (?=...) is a lookahead; a non-capture group is (?:...).
>
> The regex should be r"((?:\.\.\.)|…)", which can be simplified to just
> r"\.\.\.|…" for your use-case. (You don't need the
> re.IGNORECASE|re.VERBOSE either!)
Thanks,
I made same mistake in another place, but dont see it here.
A+
Laurent.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Can I download XML data from the web and save, in as CSV or TXT delimitation?
On Wed, 19 Aug 2015 04:57:44 -0700, ryguy7272 wrote: [stuff] Downloading xml from the web is easy writing csv or txt is easy The tricky bit is converting the xml you have into the csv or text data you want. And to do that, you need to understand the structure of the xml data that you are receiving and how the xml nodes and their attributes and values should be mapped into the csv file you want to create. Unfortunately I don't think that there is a single standard mechanism for doing that bit, although there are some tools and libraries that can help. -- Denis McMahon, denismfmcma...@gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Can I download XML data from the web and save, in as CSV or TXT delimitation?
> On Aug 19, 2015, at 7:01 PM, Denis McMahon wrote: > > Downloading xml from the web is easy > > writing csv or txt is easy > > The tricky bit is converting the xml you have into the csv or text data > you want. > curl | xml2 | 2csv http://www.ofb.net/~egnor/xml2/ref -- https://mail.python.org/mailman/listinfo/python-list
ANN: Wing IDE 5.1.6 released
Hi, Wingware has released version 5.1.6 of Wing IDE, our cross-platform integrated development environment for the Python programming language. Wing IDE features a professional code editor with vi, emacs, visual studio, and other key bindings, auto-completion, call tips, context-sensitive auto-editing, goto-definition, find uses, refactoring, a powerful debugger, version control, unit testing, search, project management, and many other features. This release includes the following improvements: Support for debugging code running on Raspberry Pi Support for debugging Python 3.5c1+ Option to run more than one test file concurrently from the Testing tool Fix several problems with Django project creation Show correct stdout/stderr output from pytest unit tests in the Testing tool Partially updated French localization (thanks to Jean Sanchez) Fix autocompletion after from . and from ..name statements Correctly reuse locked splits for already-open files Fix editing input lines in Debug I/O About 40 other improvements For details see http://wingware.com/news/2015-08-18 and http://wingware.com/pub/wingide/5.1.6/CHANGELOG.txt What's New in Wing 5.1: Wing IDE 5.1 adds multi-process and child process debugging, syntax highlighting in the shells, support for pytest, Find Symbol in Project, persistent time-stamped unit test results, auto-conversion of indents on paste, an XCode keyboard personality, support for Flask, Django 1.7 and 1.8, Python 3.5 and recent Google App Engine versions, improved auto-completion for PyQt, recursive snippet invocation, and many other minor features and improvements. Free trial: http://wingware.com/wingide/trial Downloads: http://wingware.com/downloads Feature list: http://wingware.com/wingide/features Sales: http://wingware.com/store/purchase Upgrades: https://wingware.com/store/upgrade Questions? Don't hesitate to email us at supp...@wingware.com. Thanks, -- Stephan Deibel Wingware | Python IDE The Intelligent Development Environment for Python Programmers wingware.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Can I download XML data from the web and save, in as CSV or TXT delimitation?
On Wednesday, August 19, 2015 at 1:14:44 PM UTC-4, Petite Abeille wrote: > > On Aug 19, 2015, at 7:01 PM, Denis McMahon wrote: > > > > Downloading xml from the web is easy > > > > writing csv or txt is easy > > > > The tricky bit is converting the xml you have into the csv or text data > > you want. > > > > curl | xml2 | 2csv > > http://www.ofb.net/~egnor/xml2/ref WOW!!! This is so powerful!!! https://pypi.python.org/pypi/xmlutils You're right, Denis, it is a bit tricky. I definitely need to play with this a bit to get it to do what I really want it to do, but anyway, I think I can take it from here. Thanks everyone!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Logging to a file from a C-extension
Al Pfalzgraf schrieb am 18.08.2015 um 15:07: > If a logging file is opened at the level of a Python application, how > would the log file name be communicated to a C-extension so that logging > from the extension would be sent to the same log file? Writing to the file directly (as was suggested) may not be a good idea as it would bypass the log filtering and formatting. Instead, I'd suggest sending output to a normal Python Logger object instead. This is obviously trivial in Cython (where you can just implement it in Python code), but you can do the same in C with just the usual C-API overhead. Stefan -- https://mail.python.org/mailman/listinfo/python-list
Re: Python re to extract useful information from each line
Here is a first shot at a pyparsing parser for these lines:
from pyparsing import *
SET,POLICY,ID,FROM,TO,NAT,SRC,DST,IP,PORT,SCHEDULE,LOG,PERMIT,ALLOW,DENY =
map(CaselessKeyword,
"SET,POLICY,ID,FROM,TO,NAT,SRC,DST,IP,PORT,SCHEDULE,LOG,PERMIT,ALLOW,DENY".split(','))
integer = Word(nums)
ipAddr = Combine(integer + ('.'+integer)*3)
quotedString.setParseAction(removeQuotes)
logParser = (SET + POLICY + ID + integer("id") +
FROM + quotedString("from_") +
TO + quotedString("to_") + quotedString("service"))
I run this with:
for line in """
1- set policy id 1000 from "Untrust" to "Trust" "Any" "1.1.1.1" "HTTP" nat dst
ip 10.10.10.10 port 8000 permit log
2- set policy id 5000 from "Trust" to "Untrust" "Any" "microsoft.com" "HTTP"
nat src permit schedule "14August2014" log
3- set policy id 7000 from "Trust" to "Untrust" "Users" "Any" "ANY" nat src
dip-id 4 permit log
4- set policy id 7000 from "Trust" to "Untrust" "servers" "Any" "ANY" deny
""".splitlines():
line = line.strip()
if not line: continue
print (integer + '-' + logParser).parseString(line).dump()
print
Getting:
['1', '-', 'SET', 'POLICY', 'ID', '1000', 'FROM', 'Untrust', 'TO', 'Trust',
'Any']
- from_: Untrust
- id: 1000
- service: Any
- to_: Trust
['2', '-', 'SET', 'POLICY', 'ID', '5000', 'FROM', 'Trust', 'TO', 'Untrust',
'Any']
- from_: Trust
- id: 5000
- service: Any
- to_: Untrust
['3', '-', 'SET', 'POLICY', 'ID', '7000', 'FROM', 'Trust', 'TO', 'Untrust',
'Users']
- from_: Trust
- id: 7000
- service: Users
- to_: Untrust
['4', '-', 'SET', 'POLICY', 'ID', '7000', 'FROM', 'Trust', 'TO', 'Untrust',
'servers']
- from_: Trust
- id: 7000
- service: servers
- to_: Untrust
Pyparsing adds Optional classes so that you can include expressions for pieces
that might be missing like "... + Optional(NAT + (SRC | DST)) + ..."
-- Paul
--
https://mail.python.org/mailman/listinfo/python-list
Check if dictionary empty with == {}
Probably a silly question.
Let's say I have a dictionary mydict and I need to test if a dictionary is
empty.
I would use
if not mydict:
"""do something"""
But I just came across a line of code like:
if mydict == {}:
"""do something"""
which seems odd to me, but maybe there is a valid use case, thus I decided to
ask the community.
Thanks.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Check if dictionary empty with == {}
Comparison against {} will be less efficient. You need to create a
dictionary every time instead of just checking the length of your
dictionary, which is likely stored in the header. So, it will work, but
certainly isn't idiomatic Python.
Skip
--
https://mail.python.org/mailman/listinfo/python-list
Re: Check if dictionary empty with == {}
On 2015-08-20 00:15, Skip Montanaro wrote:
Comparison against {} will be less efficient. You need to create a
dictionary every time instead of just checking the length of your
dictionary, which is likely stored in the header. So, it will work, but
certainly isn't idiomatic Python.
Well, that depends on the intention.
Is it checking whether the dictionary is empty, or whether it's an
empty dictionary (and not, say, an empty list)?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Check if dictionary empty with == {}
On 2015-08-19 15:57, Anton wrote:
> Probably a silly question.
> Let's say I have a dictionary mydict and I need to test if a
> dictionary is empty.
>
> I would use
>
> if not mydict:
> """do something"""
>
> But I just came across a line of code like:
>
> if mydict == {}:
> """do something"""
>
> which seems odd to me, but maybe there is a valid use case, thus I
> decided to ask the community.
The only valid reason is "the person who wrote that line doesn't
speak idiomatic Python", and that it should be changed to "if not
mydict" at your next code checking :-D
-tkc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Check if dictionary empty with == {}
On Wed, Aug 19, 2015 at 6:33 PM, MRAB wrote:
> Well, that depends on the intention.
>
> Is it checking whether the dictionary is empty, or whether it's an
> empty dictionary (and not, say, an empty list)?
>
Sure, that's a possibility. I would argue that "mydict == {}" is also not
the idiomatic way to see if mydict isn't an empty list. For that, you'd use
something like
if mydict or not isinstance(mydict, list):
blah blah blah
Skip
--
https://mail.python.org/mailman/listinfo/python-list
Re: JSON Object to CSV file
On Wednesday, June 17, 2015 at 11:00:24 AM UTC-4, kbtyo wrote:
> I would like to have this JSON object written out to a CSV file so that the
> keys are header fields (for each of the columns) and the values are values
> that are associated with each header field. Is there a best practice for
> working with this? Ideally I would like to recursively iterate through the
> key value pairs. Thank you in advance. I am using Python 3.4 on Windows. My
> editor is Sublime 2.
>
> {
> "CF": {
> "A": "5",
> "FEC": "1/1/0001 12:00:00 AM",
> "TE": null,
> "Locator": null,
> "Message": "Transfer Fee",
> "AT": null,
> "FT": null,
> "FR": "True",
> "FY": null,
> "FR": null,
> "FG": "0",
> "Comment": null,
> "FUD": null,
> "cID": null,
> "GEO": null,
> "ISO": null,
> "TRID": null,
> "XTY": "931083",
> "ANM": null,
> "NM": null
> },
> "CF": "Fee",
> "ID": "2"
> }
Please see this link.
https://pypi.python.org/pypi/xmlutils
I'm not sure if that will help you. I just found it toady. Also, I'm pretty
new to Python. Anyway, hopefully, it gets you going in the right direction...
--
https://mail.python.org/mailman/listinfo/python-list
Re: JSON Object to CSV Question
On Thursday, June 18, 2015 at 2:59:11 AM UTC-4, kbtyo wrote:
> Good Evening Everyone:
>
>
> I would like to have this JSON object written out to a CSV file so that the
> keys are header fields (for each of the columns) and the values are values
> that are associated with each header field. Is there a best practice for
> working with this? Ideally I would like to recursively iterate through the
> key value pairs. Thank you in advance. I am using Python 3.4 on Windows. My
> editor is Sublime 2.
>
>
>
> Here is the JSON object:
>
>
> {
> "Fee": {
> "A": "5",
> "FEC": "1/1/0001 12:00:00 AM",
> "TE": null,
> "Locator": null,
> "Message": "Transfer Fee",
> "AT": null,
> "FT": null,
> "FR": "True",
> "FY": null,
> "FR": null,
> "FG": "0",
> "Comment": null,
> "FUD": null,
> "cID": null,
> "GEO": null,
> "ISO": null,
> "TRID": null,
> "XTY": "931083",
> "ANM": null,
> "NM": null
> },
> "CF": "Fee",
> "ID": "2"
> }
>
> The value, "Fee" associated with the key, "CF" should not be included as a
> column header (only as a value of the key "CF").
>
>
>
> Other than the former, the keys should be headers and the corresponding
> tuples - the field values.
>
> In essence, my goal is to the following:
>
> You get a dictionary object (the "outer" dictionary)You get the data from the
> "CF" key (fixed name?) which is a string ("Fee" in your example)You use that
> value as a key to obtain another value from the same "outer" dictionary,
> which should be a another dictionary (the "inner" dictionary)You make a CSV
> file with:
> a header that contains "CF" plus all keys in the "inner" dictionary that have
> an associated valuethe value from key "CF" in the "outer" dictionary plus all
> non-null values in the "inner" dictionary.
> I have done the following:
>
>
> import csv
> import json
> import sys
>
> def hook(obj):
> return obj
>
> def flatten(obj):
> for k, v in obj:
> if isinstance(v, list):
> yield from flatten(v)
> else:
> yield k, v
>
> if __name__ == "__main__":
> with open("data.json") as f:
> data = json.load(f, object_pairs_hook=hook)
>
> pairs = list(flatten(data))
>
> writer = csv.writer(sys.stdout)
> writer.writerow([k for k, v in pairs])
> writer.writerow([v for k, v in pairs])
>
>
>
> The output is as follows:
>
> $ python3 json2csv.py
> A,FEC,TE,Locator,Message,AT,FT,FR,FY,FR,FG,Comment,FUD,cID,GEO,ISO,TRID,XTY,ANM,NM,CF,ID
>
> 5,1/1/0001 12:00:00 AM,,,Transfer Fee,,,True,,,0,,,931083,,,Fee,2
>
>
> I do not want to have duplicate column names.
>
>
>
> Any advice on other best practices that I may utilize?
I am literally just learning Python now...total newbie...don't take this as the
be-all-end-all...but maybe this will work for you:
https://pypi.python.org/pypi/xmlutils
Read the entire page.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Check if dictionary empty with == {}
On Thursday 20 August 2015 08:57, Anton wrote:
> Probably a silly question.
> Let's say I have a dictionary mydict and I need to test if a dictionary is
> empty.
>
> I would use
>
> if not mydict:
> """do something"""
>
> But I just came across a line of code like:
>
> if mydict == {}:
> """do something"""
>
> which seems odd to me, but maybe there is a valid use case, thus I decided
> to ask the community.
It's neither good code or bad code. It looks like something a beginner or
casual Python user (e.g. a sys admin) might have written, but that isn't
necessarily bad. It's no more wrong than writing `if x == 0` to test for an
"empty" number (or "nothing", in the numeric sense).
Pros:
+ It's pretty short and explicit.
+ It allows for duck-typing: if some de facto mapping object wants to
support the dict API without inheriting from dict, it can.
+ Even a beginner can understand it: "does mydict equal an
empty dict?"
Cons:
- It looks a bit Python 1.5-ish. People used to more modern
idioms may have an (unjustified, in my opinion) "WTF" moment
when looking at it.
- It *might* be a bit slow, since it takes time to create an
empty dict; on the other hand, it also takes time to call
isinstance(), so if you care about this, I want to see your
profiling results and benchmarks.
Actually, it's not a bit slow, it's *significantly* faster than an
isinstance check:
steve@runes:~$ python2.7 -m timeit -s "mydict = {1:2}" \
> "if mydict == {}: pass"
1000 loops, best of 3: 0.0872 usec per loop
steve@runes:~$ python2.7 -m timeit -s "mydict = {1:2}" \
> "if isinstance(mydict, dict) and not mydict: pass"
100 loops, best of 3: 0.257 usec per loop
So maybe it's a micro-optimization?
TL;DR
There's nothing wrong with it. It is ever-so-subtly different from the
various alternatives, so if you change it, don't be surprised if you break
something.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Storing dictionary locations as a string and using eval - alternatives?
Hi,
I have a textfile with a bunch of JSON objects, one per line.
I'm looking at parsing each of these, and extract some metrics from each line.
I have a dict called "metrics_to_extract", containing the metrics I'm looking
at extracting. In this, I store a name used to identify the metric, along with
the location in the parsed JSON object.
Below is my code:
>>>
metrics_to_extract = {
'current_connections': "server_status_json['connections']['current']",
'resident_memory': "server_status_json['mem']['resident']"
}
def add_point(name, value, timestamp, tags):
return {
"measurement": name,
"tags": tags,
# "time": timestamp.isoformat(),
"time": timestamp,
"fields": {
"value": float(value)
}
}
with open(input_file, 'r') as f:
json_points = []
for line in f:
if line.startswith("{"):
server_status_json = json.loads(line)
# pp.pprint(server_status_json)
# import ipdb; ipdb.set_trace()
timestamp = server_status_json['localTime']
tags = {
'project': project,
'hostname': server_status_json['host'],
'version': server_status_json['version'],
'storage_engine': server_status_json['storageEngine']['name']
}
for key, value in metrics_to_extract.items():
json_points.append(add_point(key, eval(value), timestamp, tags))
# client.write_points(json_points)
else:
print("non matching line")
>>>
My question is - I'm using "eval" in the above, with the nested location (e.g.
"server_status_json['mem']['resident']") stored as a string.
I get the feeling this isn't particularly idiomatic or a great way of doing it
- and would be keen to hear alternative suggestions?
Thanks,
Victor
--
https://mail.python.org/mailman/listinfo/python-list
Re: Storing dictionary locations as a string and using eval - alternatives?
On Thursday 20 August 2015 13:59, Victor Hooi wrote:
> Hi,
>
> I have a textfile with a bunch of JSON objects, one per line.
>
> I'm looking at parsing each of these, and extract some metrics from each
> line.
>
> I have a dict called "metrics_to_extract", containing the metrics I'm
> looking at extracting. In this, I store a name used to identify the
> metric, along with the location in the parsed JSON object.
>
> Below is my code:
>
> metrics_to_extract = {
> 'current_connections': "server_status_json['connections']['current']",
> 'resident_memory': "server_status_json['mem']['resident']"
> }
>
>
> def add_point(name, value, timestamp, tags):
> return {
> "measurement": name,
> "tags": tags,
> # "time": timestamp.isoformat(),
> "time": timestamp,
> "fields": {
> "value": float(value)
> }
> }
The function name is misleading and you should consider changing it.
Misleading names make code harder to understand and maintain. The function
doesn't "add point" at all, that implies that it should modify some data
structure by adding (appending?) a point to it. Instead, it generates a dict
containing (I presume) a data point.
But moving on...
Change metrics_to_extract above to this:
metrics_to_extract = {
'current_connections': ['connections', 'current'],
'resident_memory': ['mem', 'resident'],
}
add this helper function:
def get_nested_items(obj, *names):
"""Return obj[name][name2] ... [nameN] for any list of names."""
for name in names:
obj = obj[name]
return obj
and modify the code below as shown:
> with open(input_file, 'r') as f:
> json_points = []
> for line in f:
> if line.startswith("{"):
[...]
> for key, value in metrics_to_extract.items():
> json_points.append(add_point(key, eval(value), timestamp,
> tags))
for key, names in metrics_to_extract.items():
value = get_nested_items(server_status_json, *names)
point = add_point(key, value, timestamp, tags)
json_points.append(point)
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Storing dictionary locations as a string and using eval - alternatives?
Victor Hooi writes: > ... I did not read your message body - just your "subject". Instead of (the potentially insecure) "eval", I use "json.loads" (combined with a "json.dumps" to get the "json" representation). See the "json" module documentation. -- https://mail.python.org/mailman/listinfo/python-list
