Re: Thread getting stuck\hang
Hi Dennis, My requirement is like,i want to send the static and dynamic traffic together.while traffic sending i want to delete some config from device. st and dy both the targets want to be run at the same time. On Fri, Apr 28, 2017 at 6:10 PM, Dennis Lee Bieber wrote: > On Fri, 28 Apr 2017 14:42:33 +0530, Iranna Mathapati > declaimed the following: > > >Hi Dennis, > > > >Two threads(_st and _dy targets) sharing the same user define function at > >the same time...This is the reason for hang/stuck? > > > >in both the thread ,we are calling same user define function > >(itgen_explorer.startTrafficFlows) and greping the values from return > >function ,using it in main function. > > > > While the threads will have their own stacks, meaning function > local > data should be separate, if that common function is written anything like > the st/dy threads (that is, full of "global" declarations) anything could > be happening. > > > Given that your original sample main logic basically just started > three > threads and almost immediately waits for them to complete by using .join(), > I have to wonder just why you are using threads at all. > > Due to the GIL, even on a multi-core processor, only one thread can > actually do anything at a time (threads work great when they spend most of > their time waiting for I/O operations to complete, but are terrible if > trying to do heavy number crunching). If you don't have lots of blocking > I/O, the thread switching will actually make using threads take more time > then just calling the functions one after the other. > > From your original post, what happens if you replace: > > >sniffer1 = > >threading.Thread(target=validate_traffic_stats_dy, > args=(FT_item_dy,RT_item_dy,forward_path_list_dy,return_ > path_list_dy,nat_type_list_dy,pkt_dy_1,)) > >sniffer2 = > >threading.Thread(target=validate_traffic_stats_st, > args=(FT_item_st,RT_item_st,forward_path_list_st,return_ > path_list_st,nat_type_list_st,pkt_st,)) > >sniffer1.start() > >sniffer2.start() > >sniffer3 = threading.Thread(target=delete_new_dynamic_nat_conf) > >sniffer3.start() > >sniffer1.join() > >sniffer2.join() > >sniffer3.join() > > with just the inline calls: > > validate_traffic_stats_dy(the arg list) > validate_traffic_stats_st(its arg list) > delete_new_dynamic_nat_conf() > > That should run them in sequence, with no chance for conflicts > between > shared state. > > If you still get erratic hanging, you have a logic problem > somewhere > else in the program other than a shared state deadlock. > -- > Wulfraed Dennis Lee Bieber AF6VN > wlfr...@ix.netcom.comHTTP://wlfraed.home.netcom.com/ > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Convert text file data into RDF format through the Python
Hi, I would like to ask how can I convert text file data into RDF fromat. data look like this: # sent_id = weblog-juancole.com_juancole_20051126063000_ENG_20051126_063000-0001 # text = Al-Zaman : American forces killed Shaikh Abdullah al-Ani, the preacher at the mosque in the town of Qaim, near the Syrian border. 1 Al Al PROPN NNP Number=Sing 0 root_ SpaceAfter=No 2 - - PUNCT HYPH_ 1 punct _ SpaceAfter=No 3 Zaman Zaman PROPN NNP Number=Sing 1 flat_ _ 4 : : PUNCT : _ 1 punct _ _ 5 AmericanamericanADJ JJ Degree=Pos 6 amod_ _ please suggest me -- https://mail.python.org/mailman/listinfo/python-list
Inconsistency between dict() and collections.OrderedDict() methods.
I have a subclass of dict that enforces which keys are allowed to be set
and only allows each key to be set at most once:
class StrictDict(dict):
def __init__(self, validkeys, *args, **kwargs):
self.validkeys = validkeys
super(StrictDict, self).__init__(*args, **kwargs)
def __setitem__(self, key, value):
if key not in self.validkeys:
raise KeyError("'%s' is not a valid key" % key)
if key in self.keys():
raise KeyError("'%s' is already set" % key)
super(StrictDict, self).__setitem__(key, value)
This works fine in the general case:
/tmp$ ~/sw/Python-3.6.1/python
Python 3.6.1 (heads/master:7bcbcb9, Apr 21 2017, 01:52:28)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from strictdict import *
>>> d = StrictDict(('foo', 'bar'))
>>> d['foo'] = 1
>>> d['foo'] = 1
Traceback (most recent call last):
File "", line 1, in
File "/tmp/strictdict.py", line 11, in __setitem__
raise KeyError("'%s' is already set" % key)
KeyError: "'foo' is already set"
>>> d['baz'] = 1
Traceback (most recent call last):
File "", line 1, in
File "/tmp/strictdict.py", line 8, in __setitem__
raise KeyError("'%s' is not a valid key" % key)
KeyError: "'baz' is not a valid key"
>>> d
{'foo': 1}
However, when I call other methods which set items in the dictionary, I
do not get errors for invalid or duplicate keys (__setitem__ is not called):
/tmp$ ~/sw/Python-3.6.1/python
Python 3.6.1 (heads/master:7bcbcb9, Apr 21 2017, 01:52:28)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from strictdict import *
>>> d = StrictDict(('foo', 'bar'), baz=1)
>>> d
{'baz': 1}
>>> d.update({'spam': 1})
>>> d
{'baz': 1, 'spam': 1}
It seems a little onerous that I have to put the key checks in several
places and implement each of those APIs manually again (and keep on top
of that if dict() grows some new methods that involve setting items). Is
there a compelling reason why the dict module doesn't call a custom
__setitem__ each time an item needs to be set? Performance is
undoubtably a concern with something a fundamental as dicts, but what
I'm suggesting could be done fairly efficiently (with a single "is the
self object a subclass which overrides __setitem__" test followed by
running the existing code or a slower version which calls __setitem__).
Anyway, related to but separate from that, as the subject of this
message states it turns out that collections.OrderedDict() *does* do
what I would expect:
from collections import OrderedDict
class OrderedStrictDict(OrderedDict):
def __init__(self, validkeys, *args, **kwargs):
self.validkeys = validkeys
super(OrderedStrictDict, self).__init__(*args, **kwargs)
def __setitem__(self, key, value):
if key not in self.validkeys:
raise KeyError("'%s' is not a valid key" % key)
if key in self.keys():
raise KeyError("'%s' is already set" % key)
super(OrderedStrictDict, self).__setitem__(key, value)
/tmp$ ~/sw/Python-3.6.1/python
Python 3.6.1 (heads/master:7bcbcb9, Apr 21 2017, 01:52:28)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from strictdict import *
>>> d = OrderedStrictDict(('foo', 'bar'), baz=1)
Traceback (most recent call last):
File "", line 1, in
File "/tmp/strictdict.py", line 19, in __init__
super(OrderedStrictDict, self).__init__(*args, **kwargs)
File "/tmp/strictdict.py", line 23, in __setitem__
raise KeyError("'%s' is not a valid key" % key)
KeyError: "'baz' is not a valid key"
>>> d = OrderedStrictDict(('foo', 'bar'))
>>> d.update({'baz': 1})
Traceback (most recent call last):
File "", line 1, in
File "/tmp/strictdict.py", line 23, in __setitem__
raise KeyError("'%s' is not a valid key" % key)
KeyError: "'baz' is not a valid key"
The documentation for OrderedDict says: "Return an instance of a dict
subclass, supporting the usual dict methods. An OrderedDict is a dict
that remembers the order that keys were first inserted. If a new entry
overwrites an existing entry, the original insertion position is left
unchanged. Deleting an entry and reinserting it will move it to the end."
The strong implication there is that the class behaves exactly like a
dict apart from the ordering, but that's not true.
E.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
On Saturday, April 29, 2017 at 4:20:06 PM UTC-4, Erik wrote: > It seems a little onerous that I have to put the key checks in several > places and implement each of those APIs manually again (and keep on top > of that if dict() grows some new methods that involve setting items). Is > there a compelling reason why the dict module doesn't call a custom > __setitem__ each time an item needs to be set? Performance is > undoubtably a concern with something a fundamental as dicts... Performance is the reason. For creating your own class that acts like a dict, you should derive from collections.abc.MutableMapping, which only requires implementing __getitem__, __setitem__, __delitem__, __iter__, and __len__. --Ned. -- https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
On 29/04/17 23:40, Ned Batchelder wrote: For creating your own class that acts like a dict, you should derive from collections.abc.MutableMapping, which only requires implementing __getitem__, __setitem__, __delitem__, __iter__, and __len__. Or, I could derive from collections.OrderedDict and just implement the two methods that I actually want to change the behavior of (did you read the rest of my post?) ;) That's one of the points I'm trying to make - why is it harder than it needs to be to do something this simple? The other is that the documentation of collections.OrderedDict seems to be lacking (it is talking in terms of being a "dict" subclass, but it actually isn't one). E. -- https://mail.python.org/mailman/listinfo/python-list
Not understanding itertools.dropwhile()
< start code > import itertools data = """Line1 Line2 Line4 Line5""" def test_to_start(s): return "2" in s for line in itertools.dropwhile(test_to_start, data.splitlines()): print(line) < end code > I expect: $ python3 dropwhile.py Line2 Line4 Line5 I get: $ python3 dropwhile.py Line1 Line2 Line4 Line5 Please explain. -- https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
Erik wrote: That's one of the points I'm trying to make - why is it harder than it needs to be to do something this simple? The built-in dict class is used internally to implement various namespaces (module, class, instance, etc.), so it needs to be extremely efficient. Funnelling all updates through a single method would be unacceptable. This does make it harder to override behaviour in a subclass, but that's always a tricky proposition that relies on knowing a lot about implementation details of the class. If a particular method isn't documented as being a hook intended for overriding, you can't make any assumptions. The other is that the documentation of collections.OrderedDict seems to be lacking (it is talking in terms of being a "dict" subclass, but it actually isn't one). The docs probably mean to say that it has the same interface as a dict, not that it's necessarily implemented as a subclass of dict. Nevertheless, it could actually be a dict subclass that overrides all the necessary methods. Either way, its behaviour under subclassing is not spelled out. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
Erik writes: > On 29/04/17 23:40, Ned Batchelder wrote: > > For creating your own class that acts like a dict, you should derive > > from collections.abc.MutableMapping, which only requires > > implementing __getitem__, __setitem__, __delitem__, __iter__, and > > __len__. > > Or, I could derive from collections.OrderedDict and just implement the > two methods that I actually want to change the behavior of (did you > read the rest of my post?) ;) Did you read Ned's? :-) You say that there are only two methods you want to change the behaviour of; but as you have found, those two methods are not the only ones you need to implement, in order to get the changed behaviour. > That's one of the points I'm trying to make - why is it harder than it > needs to be to do something this simple? The “needs” have already been expressed, by yourself and Ned: the type needs to provide fast performance. One cost of that, with the current implementation, is that you need to implement the MutableMapping protocol if you want a custom class. -- \“Perchance you who pronounce my sentence are in greater fear | `\ than I who receive it.” —Giordano Bruno, burned at the stake by | _o__) the Catholic church for the heresy of heliocentrism, 1600-02-16 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
On Sunday, April 30, 2017 at 12:23:19 AM UTC+1, Erik wrote: > On 29/04/17 23:40, Ned Batchelder wrote: > > For creating your own class that acts like > > a dict, you should derive from collections.abc.MutableMapping, which > > only requires implementing __getitem__, __setitem__, __delitem__, > > __iter__, and __len__. > > Or, I could derive from collections.OrderedDict and just implement the > two methods that I actually want to change the behavior of (did you read > the rest of my post?) ;) > > That's one of the points I'm trying to make - why is it harder than it > needs to be to do something this simple? > > The other is that the documentation of collections.OrderedDict seems to > be lacking (it is talking in terms of being a "dict" subclass, but it > actually isn't one). > > E. Could have fooled me. C:\python Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from collections import OrderedDict >>> o = OrderedDict() >>> isinstance(o, dict) True Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
tempname.mktemp functionality deprecation
Working on some deduplication code, I want do my my best at
performing an atomic re-hard-linking atop an existing file, akin to
"ln -f source.txt dest.txt"
However, when I issue
os.link("source.txt", "dest.txt")
it fails with an OSError (EEXISTS). This isn't surprising as it's
documented. Unfortunately, os.link doesn't support something like
os.link("source.txt", "dest.txt", force=True)
However, I don't want to
os.unlink("dest.txt")
os.link("source.txt", "dest.txt")
in the event the power goes out between the unlink() and the link(),
leaving me in a state where dest.txt is deleted but the link hasn't
yet happened.
So my plan was to do something like
temp_name = tempfile.mktemp(dir=DIRECTORY_CONTAINING_SOURCE_TXT)
os.link("source.txt", temp_name)
try:
os.rename(temp_name, "dest.txt") # docs guarantee this is atomic
except OSError:
os.unlink(temp_name)
There's still the potential leakage if a crash occurs, but I'd rather
have an extra hard-link floating around than lose an original
file-name.
Unfortunately, tempfile.mktemp() is described as deprecated
since 2.3 (though appears to still exist in the 3.4.2 that is the
default Py3 on Debian Stable). While the deprecation notice says
"In version 2.3 of Python, this module was overhauled for enhanced
security. It now provides three new functions, NamedTemporaryFile(),
mkstemp(), and mkdtemp(), which should eliminate all remaining need
to use the insecure mktemp() function", as best I can tell, all of
the other functions/objects in the tempfile module return a file
object, not a string suitable for passing to link().
So which route should I pursue?
- go ahead and use tempfile.mktemp() ignoring the deprecation?
- use a GUID-named temp-file instead for less chance of collision?
- I happen to already have a hash of the file contents, so use
the .hexdigest() string as the temp-file name?
- some other solution I've missed?
Thanks,
-tkc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Not understanding itertools.dropwhile()
On 4/29/2017 7:41 PM, Jason Friedman wrote: < start code > import itertools data = """Line1 Line2 Line4 Line5""" def test_to_start(s): return "2" in s for line in itertools.dropwhile(test_to_start, data.splitlines()): print(line) < end code > I expect: $ python3 dropwhile.py Line2 Line4 Line5 I get: $ python3 dropwhile.py Line1 Line2 Line4 Line5 Please explain. '2' is not in the first line, so 'dropping' stops. Read the equivalent generator code in the doc. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
On 30/04/17 01:31, Ben Finney wrote: Erik writes: On 29/04/17 23:40, Ned Batchelder wrote: For creating your own class that acts like a dict, you should derive from collections.abc.MutableMapping, which only requires implementing __getitem__, __setitem__, __delitem__, __iter__, and __len__. Or, I could derive from collections.OrderedDict and just implement the two methods that I actually want to change the behavior of (did you read the rest of my post?) ;) Did you read Ned's? :-) Yes, I did. Which part of Ned's do you think I did not read properly? You say that there are only two methods you want to change the behaviour of; Yes, __init__ (so that I can store the valid keys) and __setitem__ (so that I can check the key being set). but as you have found, those two methods are not the only ones you need to implement, in order to get the changed behaviour. I didn't find that. But I found that I could change exactly those two methods if I just derived from a different class that also provides another behavior that I don't need (ordered keys). you need to implement the MutableMapping protocol if you want a custom class Why do I "need" to do that when I could subclass from OrderedDict instead? If I'm not being clear: Is there not a class that is somewhere between "dict" and "OrderedDict" that provides what I need? collections.abc.MutableMapping is not that class, as it requires me to write all sorts of methods that are irrelevant to the behavior changes I want to make (__len__ etc). E. -- https://mail.python.org/mailman/listinfo/python-list
Re: Not understanding itertools.dropwhile()
On Sun, 30 Apr 2017 09:41 am, Jason Friedman wrote:
> < start code >
>
> import itertools
>
> data = """Line1
> Line2
>
> Line4
> Line5"""
>
> def test_to_start(s):
> return "2" in s
>
> for line in itertools.dropwhile(test_to_start, data.splitlines()):
> print(line)
>
> < end code >
>
> I expect:
>
> $ python3 dropwhile.py
> Line2
>
> Line4
> Line5
dropwhile drops lines while the condition is true. Once the condition is
false, it stops dropping lines and returns them all. Since the condition
you give is not true at the start, nothing is dropped.
This example may explain how dropwhile is used:
py> lines = """line 1
... line 2
... line 3
... line 4
... another line
... and another line again
... and more lines
... line 8
... line 9
... and one more""".splitlines()
py> def startswith(text):
... return text.startswith("line")
...
py> for line in itertools.dropwhile(startswith, lines):
... print(line)
...
another line
and another line again
and more lines
line 8
line 9
and one more
Only lines at the start of the sequence will be dropped. Once the condition
becomes false, everything else is passed through. We can implement our own
version of dropwhile like this:
def my_dropwhile(condition, iterable):
it = iter(iterable)
for item in it:
if not condition(item):
yield item
break
for item in it: # anything remaining
yield item
If you just want to remove lines, whether they are at the start or the
middle or the end, you should use either the built-in filter() or
itertools.filterfalse().
--
Steve
Emoji: a small, fuzzy, indistinct picture used to replace a clear and
perfectly comprehensible word.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
On 30/04/17 01:17, breamore...@gmail.com wrote: On Sunday, April 30, 2017 at 12:23:19 AM UTC+1, Erik wrote: The other is that the documentation of collections.OrderedDict seems to be lacking (it is talking in terms of being a "dict" subclass, but it actually isn't one). E. Could have fooled me. C:\python Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. from collections import OrderedDict o = OrderedDict() isinstance(o, dict) True OK, so technically it is a subclass, I didn't notice that, but where does the documentation explain all of the differences in behaviour? It doesn't. It states "it's a subclass of dict that does and " (by which I mean "order the keys") and it's not unreasonable to therefore believe that's _all_ it does differently. If if it actually does ", , , , " (by which I mean "order the keys and implement very different behavior for other methods such as __init__ and __update__") then the documentation is lacking (because it doesn't mention it). Isn't that what I said? E. -- https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
On Sunday, April 30, 2017 at 2:30:25 AM UTC+1, Erik wrote: > On 30/04/17 01:17, breamoreboy wrote: > > On Sunday, April 30, 2017 at 12:23:19 AM UTC+1, Erik wrote: > >> The other is that the documentation of collections.OrderedDict seems to > >> be lacking (it is talking in terms of being a "dict" subclass, but it > >> actually isn't one). > >> > >> E. > > > > Could have fooled me. > > > > C:\python > > Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit > > (AMD64)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > from collections import OrderedDict > o = OrderedDict() > isinstance(o, dict) > > True > > OK, so technically it is a subclass, I didn't notice that, but where > does the documentation explain all of the differences in behaviour? It > doesn't. It states "it's a subclass of dict that does and " > (by which I mean "order the keys") and it's not unreasonable to > therefore believe that's _all_ it does differently. If if it actually > does ", , , , " (by which I mean > "order the keys and implement very different behavior for other methods > such as __init__ and __update__") then the documentation is lacking > (because it doesn't mention it). Isn't that what I said? > > E. So provide a documentation patch on the bug tracker. Once you've done that how about subclassing UserDict https://docs.python.org/3.5/library/collections.html#collections.UserDict, pinching any code from OrderedDict as needed? Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
Re: tempname.mktemp functionality deprecation
On Sat, Apr 29, 2017 at 11:45 AM, Tim Chase wrote: > Unfortunately, tempfile.mktemp() is described as deprecated > since 2.3 (though appears to still exist in the 3.4.2 that is the > default Py3 on Debian Stable). While the deprecation notice says > "In version 2.3 of Python, this module was overhauled for enhanced > security. It now provides three new functions, NamedTemporaryFile(), > mkstemp(), and mkdtemp(), which should eliminate all remaining need > to use the insecure mktemp() function", as best I can tell, all of > the other functions/objects in the tempfile module return a file > object, not a string suitable for passing to link(). > > So which route should I pursue? > > - go ahead and use tempfile.mktemp() ignoring the deprecation? > > - use a GUID-named temp-file instead for less chance of collision? > > - I happen to already have a hash of the file contents, so use > the .hexdigest() string as the temp-file name? > > - some other solution I've missed? I vote the last one: you can read the .name attribute of the returned file(-like) object from NamedTemporaryFile to get a path to a file, which can be passed to other functions. I guess ideally, one would use linkat instead of os.link[*], but that's platform-specific and not exposed in Python AFAIK. Maybe things would be better if all the functions that accept filenames should also accept files, and do the best job they can? (if a platform supports using the fd instead, use that, otherwise use f.name). .. *: http://stackoverflow.com/questions/17127522/create-a-hard-link-from-a-file-handle-on-unix/18644492#18644492 -- Devin -- https://mail.python.org/mailman/listinfo/python-list
Re: tempname.mktemp functionality deprecation
On Sun, Apr 30, 2017 at 1:51 PM, Devin Jeanpierre wrote: > I guess ideally, one would use linkat instead of os.link[*], but that's > platform-specific and not exposed in Python AFAIK. It is actually - src_dir_fd and dst_dir_fd. https://docs.python.org/3/library/os.html#os.link ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: tempname.mktemp functionality deprecation
Devin Jeanpierre wrote: I vote the last one: you can read the .name attribute of the returned file(-like) object from NamedTemporaryFile to get a path to a file, which can be passed to other functions. I don't think that helps. You would have to delete the file first before you could create a link with that name, and that would leave a window of opportunity for another process to create something with the same name. I would generate a name that's likely to be unique (using mktemp() or otherwise) and try to create a link with that name. If it fails because the name is in use, generate another name and try again. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Inconsistency between dict() and collections.OrderedDict() methods.
Erik wrote: Is there not a class that is somewhere between "dict" and "OrderedDict" that provides what I need? Such a class could exist, but the stdlib doesn't happen to provide one as far as I know. Note, though, that you're relying on implementation details of OrderedDict when you use it to "fix" your problem in this way. It happens to work in some versions of Python, but it might not work in the next one. > OK, so technically it is a subclass, I didn't notice that, but where > does the documentation explain all of the differences in behaviour? It doesn't, but the differences you're talking about are differences between undocumented behavour of dict and undocumented behaviour of OrderedDict. The docs are under no obligation to tell you about any of that. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
