Cannot get the value from dogpile.cache from different modules.
1. How do I create a global variable that can be accessed by all classes?
2. I am using `dogpile.cache` to store data in the cache [1], but if I set and
get the same key from different modules, I don't get the value. Here is an
example in [2]. The value than I get is `NO_VALUE.NO_VALUE`. Why this happens?
setter is the setter.py
getter is the getter.py
Memoize is the file in [1].
[1] my dogpile class `Memoize.py`
from dogpile.cache import make_region
region = make_region().configure('dogpile.cache.memory')
def save(key, value):
"""
general purpose method to save data (value) in the cache
:param key (string) key of the value to be saved in cache
:param value (any type) the value to be saved
"""
region.set(key, value)
def get(key):
"""
general purpose method to get data from the cache
:param key (string) key of the data to be fetched
:return value (any type) data to be returned from the cache
"""
return region.get(key)
[2] My python example
`setter.py`
def myset(value):
Memoize.save("myvalue", value)
`getter.py`
def myget():
return Memoize.get("myvalue") <- this value is NO_VALUE. NO_VALUE
My class:
setter.myset(123)
getter.myget()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Cannot get the value from dogpile.cache from different modules.
On Tuesday, December 29, 2015 at 11:16:10 AM UTC, xeon Mailinglist wrote:
> 1. How do I create a global variable that can be accessed by all classes?
>
> 2. I am using `dogpile.cache` to store data in the cache [1], but if I set
> and get the same key from different modules, I don't get the value. Here is
> an example in [2]. The value than I get is `NO_VALUE.NO_VALUE`. Why this
> happens?
>
> setter is the setter.py
> getter is the getter.py
> Memoize is the file in [1].
>
>
> [1] my dogpile class `Memoize.py`
>
> from dogpile.cache import make_region
>
> region = make_region().configure('dogpile.cache.memory')
>
> def save(key, value):
> """
> general purpose method to save data (value) in the cache
>
> :param key (string) key of the value to be saved in cache
> :param value (any type) the value to be saved
> """
> region.set(key, value)
>
>
> def get(key):
> """
> general purpose method to get data from the cache
>
> :param key (string) key of the data to be fetched
> :return value (any type) data to be returned from the cache
> """
> return region.get(key)
>
>
> [2] My python example
>
> `setter.py`
>
> def myset(value):
> Memoize.save("myvalue", value)
>
> `getter.py`
>
>def myget():
> return Memoize.get("myvalue") <- this value is NO_VALUE. NO_VALUE
>
> My class:
>
> setter.myset(123)
> getter.myget()
The idea that I get from dogpile, is that in each module (getter.py, or
setter.py) there is a dictionary where the values are stored in the backend.
Hence, getter.py has its dictionary and setter.py has its dictionary also. In
the end, there is not a single dictionary where all the values should be put.
And I want a single dictionary.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Cannot get the value from dogpile.cache from different modules.
On Tuesday, December 29, 2015 at 4:18:10 PM UTC, Peter Otten wrote:
> xeon Mailinglist wrote:
>
> > On Tuesday, December 29, 2015 at 11:16:10 AM UTC, xeon Mailinglist wrote:
> >> 1. How do I create a global variable that can be accessed by all classes?
> >>
> >> 2. I am using `dogpile.cache` to store data in the cache [1], but if I
> >> set and get the same key from different modules, I don't get the value.
> >> Here is an example in [2]. The value than I get is `NO_VALUE.NO_VALUE`.
> >> Why this happens?
>
> >> region = make_region().configure('dogpile.cache.memory')
>
> The memory backend wraps a python dict whose contents are only available to
> a single script and forgotten when that script ends.
>
> My crystal ball tells me that you want to communicate between processes
> rather than "modules" and need a backend that implements persistence.
> "dogpile.cache.file" seems to be the one without dependencies outside the
> standard library.
No. My problem is that I have method1() that calls method2() which calls
myset(). method1() -> method2() -> myset(5). My problem is that, if I try to
get the value of myset() inside method1(), I can't have it. It seems that the
program has lost the value.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Cannot get the value from dogpile.cache from different modules.
On Tuesday, December 29, 2015 at 5:15:24 PM UTC, Mark Lawrence wrote:
> On 29/12/2015 15:20, xeon Mailinglist wrote:
> > On Tuesday, December 29, 2015 at 11:16:10 AM UTC, xeon Mailinglist wrote:
> >> 1. How do I create a global variable that can be accessed by all classes?
> >>
> >> 2. I am using `dogpile.cache` to store data in the cache [1], but if I set
> >> and get the same key from different modules, I don't get the value. Here
> >> is an example in [2]. The value than I get is `NO_VALUE.NO_VALUE`. Why
> >> this happens?
> >>
> >> setter is the setter.py
> >> getter is the getter.py
> >> Memoize is the file in [1].
> >>
> >>
> >> [1] my dogpile class `Memoize.py`
> >>
> >> from dogpile.cache import make_region
> >>
> >> region = make_region().configure('dogpile.cache.memory')
> >>
> >> def save(key, value):
> >>"""
> >>general purpose method to save data (value) in the cache
> >>
> >>:param key (string) key of the value to be saved in cache
> >>:param value (any type) the value to be saved
> >>"""
> >>region.set(key, value)
> >>
> >>
> >> def get(key):
> >>"""
> >>general purpose method to get data from the cache
> >>
> >>:param key (string) key of the data to be fetched
> >>:return value (any type) data to be returned from the cache
> >>"""
> >>return region.get(key)
> >>
> >>
> >> [2] My python example
> >>
> >> `setter.py`
> >>
> >> def myset(value):
> >>Memoize.save("myvalue", value)
> >>
> >> `getter.py`
> >>
> >> def myget():
> >>return Memoize.get("myvalue") <- this value is NO_VALUE. NO_VALUE
> >>
> >> My class:
> >>
> >> setter.myset(123)
> >> getter.myget()
> >
> > The idea that I get from dogpile, is that in each module (getter.py, or
> > setter.py) there is a dictionary where the values are stored in the
> > backend. Hence, getter.py has its dictionary and setter.py has its
> > dictionary also. In the end, there is not a single dictionary where all the
> > values should be put. And I want a single dictionary.
> >
>
> Then put everything in one file. Three files for the amount of code you
> show above is nonsensical. You might like to read
> http://dirtsimple.org/2004/12/python-is-not-java.html and in response to
> that http://dirtsimple.org/2004/12/java-is-not-python-either.html
>
> --
> My fellow Pythonistas, ask not what our language can do for you, ask
> what you can do for our language.
>
> Mark Lawrence
No, that's not the answer. This is just an example. I will not give you the
full code because it doesn't make any sense... I am trying to explain what is
my problem, even though I cannot reproduce it in any simple example.
I save several keys with the dogpile and everything is ok, but when I try to
save that particular value, it seems that is going to store in a new
dictionary, and not in the dictionary with all the values.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Cannot get the value from dogpile.cache from different modules.
On Tuesday, December 29, 2015 at 5:15:24 PM UTC, Mark Lawrence wrote:
> On 29/12/2015 15:20, xeon Mailinglist wrote:
> > On Tuesday, December 29, 2015 at 11:16:10 AM UTC, xeon Mailinglist wrote:
> >> 1. How do I create a global variable that can be accessed by all classes?
> >>
> >> 2. I am using `dogpile.cache` to store data in the cache [1], but if I set
> >> and get the same key from different modules, I don't get the value. Here
> >> is an example in [2]. The value than I get is `NO_VALUE.NO_VALUE`. Why
> >> this happens?
> >>
> >> setter is the setter.py
> >> getter is the getter.py
> >> Memoize is the file in [1].
> >>
> >>
> >> [1] my dogpile class `Memoize.py`
> >>
> >> from dogpile.cache import make_region
> >>
> >> region = make_region().configure('dogpile.cache.memory')
> >>
> >> def save(key, value):
> >>"""
> >>general purpose method to save data (value) in the cache
> >>
> >>:param key (string) key of the value to be saved in cache
> >>:param value (any type) the value to be saved
> >>"""
> >>region.set(key, value)
> >>
> >>
> >> def get(key):
> >>"""
> >>general purpose method to get data from the cache
> >>
> >>:param key (string) key of the data to be fetched
> >>:return value (any type) data to be returned from the cache
> >>"""
> >>return region.get(key)
> >>
> >>
> >> [2] My python example
> >>
> >> `setter.py`
> >>
> >> def myset(value):
> >>Memoize.save("myvalue", value)
> >>
> >> `getter.py`
> >>
> >> def myget():
> >>return Memoize.get("myvalue") <- this value is NO_VALUE. NO_VALUE
> >>
> >> My class:
> >>
> >> setter.myset(123)
> >> getter.myget()
> >
> > The idea that I get from dogpile, is that in each module (getter.py, or
> > setter.py) there is a dictionary where the values are stored in the
> > backend. Hence, getter.py has its dictionary and setter.py has its
> > dictionary also. In the end, there is not a single dictionary where all the
> > values should be put. And I want a single dictionary.
> >
>
> Then put everything in one file. Three files for the amount of code you
> show above is nonsensical. You might like to read
> http://dirtsimple.org/2004/12/python-is-not-java.html and in response to
> that http://dirtsimple.org/2004/12/java-is-not-python-either.html
>
> --
> My fellow Pythonistas, ask not what our language can do for you, ask
> what you can do for our language.
>
> Mark Lawrence
Here is the full class that I use to store the data.
from dogpile.cache import make_region
# my_dictionary = {}
region = make_region().configure('dogpile.cache.memory')
# arguments={"cache_dict":my_dictionary})
class Cache:
@staticmethod
def save(key, value):
"""
general purpose method to save data (value) in the cache
:param key (string) key of the value to be saved in cache
:param value (any type) the value to be saved
"""
region.set(key, value)
@staticmethod
def get(key):
"""
general purpose method to get data from the cache
:param key (string) key of the data to be fetched
:return value (any type) data to be returned from the cache
"""
return region.get(key)
@staticmethod
def get_or_create(key):
"""
General purpose method to get data from the cache. If the value does
not exist, it creates a list
:param: key (string) key of the data to be fetched
:return value (any type) data to be returned from the cache
"""
return region.get_or_create(key, list)
@staticmethod
def set_job_predictions(rank_list):
Cache.save("job_predictions", rank_list)
@staticmethod
def get_job_predictions():
return Cache.get("job_predictions")
--
https://mail.python.org/mailman/listinfo/python-list
Re: Cannot get the value from dogpile.cache from different modules.
On Tuesday, December 29, 2015 at 5:33:43 PM UTC, Peter Otten wrote:
> xeon Mailinglist wrote:
>
> > On Tuesday, December 29, 2015 at 4:18:10 PM UTC, Peter Otten wrote:
> >> xeon Mailinglist wrote:
> >>
> >> > On Tuesday, December 29, 2015 at 11:16:10 AM UTC, xeon Mailinglist
> >> > wrote:
> >> >> 1. How do I create a global variable that can be accessed by all
> >> >> classes?
> >> >>
> >> >> 2. I am using `dogpile.cache` to store data in the cache [1], but if I
> >> >> set and get the same key from different modules, I don't get the
> >> >> value. Here is an example in [2]. The value than I get is
> >> >> `NO_VALUE.NO_VALUE`. Why this happens?
> >>
> >> >> region = make_region().configure('dogpile.cache.memory')
> >>
> >> The memory backend wraps a python dict whose contents are only available
> >> to a single script and forgotten when that script ends.
> >>
> >> My crystal ball tells me that you want to communicate between processes
> >> rather than "modules" and need a backend that implements persistence.
> >> "dogpile.cache.file" seems to be the one without dependencies outside the
> >> standard library.
> >
> >
> > No.
>
> Does "No" mean "I have run my code with another backend, and the modified
> script showed the same behaviour"?
>
> > My problem is that I have method1() that calls method2() which calls
> > myset(). method1() -> method2() -> myset(5). My problem is that, if I try
> > to get the value of myset() inside method1(), I can't have it. It seems
> > that the program has lost the value.
>
> I can't make sense of that. You should be able to nest methods to your
> heart's content (as long as you don't reach the recursion limit).
>
> Can you post minimal versions of your modules in such a way that I can
> easily run them over here?
>
> If you have only one process you probably have somehow managed to get two
> backend dicts. Unfortunately there's a blind spot on my crystal ball, and I
> can't see how exactly you did it...
No, I cannot get a simpler example. The simpler example works, and in my code,
it doesn't. I thought that it was something related to the variable `region`,
but I declare it as global. So, I think that all the sets will go to the same
variable.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Cannot get the value from dogpile.cache from different modules.
On Tuesday, December 29, 2015 at 5:38:17 PM UTC, xeon Mailinglist wrote:
> On Tuesday, December 29, 2015 at 5:33:43 PM UTC, Peter Otten wrote:
> > xeon Mailinglist wrote:
> >
> > > On Tuesday, December 29, 2015 at 4:18:10 PM UTC, Peter Otten wrote:
> > >> xeon Mailinglist wrote:
> > >>
> > >> > On Tuesday, December 29, 2015 at 11:16:10 AM UTC, xeon Mailinglist
> > >> > wrote:
> > >> >> 1. How do I create a global variable that can be accessed by all
> > >> >> classes?
> > >> >>
> > >> >> 2. I am using `dogpile.cache` to store data in the cache [1], but if I
> > >> >> set and get the same key from different modules, I don't get the
> > >> >> value. Here is an example in [2]. The value than I get is
> > >> >> `NO_VALUE.NO_VALUE`. Why this happens?
> > >>
> > >> >> region = make_region().configure('dogpile.cache.memory')
> > >>
> > >> The memory backend wraps a python dict whose contents are only available
> > >> to a single script and forgotten when that script ends.
> > >>
> > >> My crystal ball tells me that you want to communicate between processes
> > >> rather than "modules" and need a backend that implements persistence.
> > >> "dogpile.cache.file" seems to be the one without dependencies outside the
> > >> standard library.
> > >
> > >
> > > No.
> >
> > Does "No" mean "I have run my code with another backend, and the modified
> > script showed the same behaviour"?
> >
> > > My problem is that I have method1() that calls method2() which calls
> > > myset(). method1() -> method2() -> myset(5). My problem is that, if I try
> > > to get the value of myset() inside method1(), I can't have it. It seems
> > > that the program has lost the value.
> >
> > I can't make sense of that. You should be able to nest methods to your
> > heart's content (as long as you don't reach the recursion limit).
> >
> > Can you post minimal versions of your modules in such a way that I can
> > easily run them over here?
> >
> > If you have only one process you probably have somehow managed to get two
> > backend dicts. Unfortunately there's a blind spot on my crystal ball, and I
> > can't see how exactly you did it...
>
> No, I cannot get a simpler example. The simpler example works, and in my
> code, it doesn't. I thought that it was something related to the variable
> `region`, but I declare it as global. So, I think that all the sets will go
> to the same variable.
Strangely enough, when I put the set values in the method1(), it works ok. Is
it because method2() is in a submodule of method1?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Cannot get the value from dogpile.cache from different modules.
On Tuesday, December 29, 2015 at 7:23:40 PM UTC, Mark Lawrence wrote:
> On 29/12/2015 17:27, xeon Mailinglist wrote:
> > On Tuesday, December 29, 2015 at 5:15:24 PM UTC, Mark Lawrence wrote:
> >> On 29/12/2015 15:20, xeon Mailinglist wrote:
> >>> On Tuesday, December 29, 2015 at 11:16:10 AM UTC, xeon Mailinglist wrote:
> >>>> 1. How do I create a global variable that can be accessed by all classes?
> >>>>
> >>>> 2. I am using `dogpile.cache` to store data in the cache [1], but if I
> >>>> set and get the same key from different modules, I don't get the value.
> >>>> Here is an example in [2]. The value than I get is `NO_VALUE.NO_VALUE`.
> >>>> Why this happens?
> >>>>
> >>>> setter is the setter.py
> >>>> getter is the getter.py
> >>>> Memoize is the file in [1].
> >>>>
> >>>>
> >>>> [1] my dogpile class `Memoize.py`
> >>>>
> >>>> from dogpile.cache import make_region
> >>>>
> >>>> region = make_region().configure('dogpile.cache.memory')
> >>>>
> >>>> def save(key, value):
> >>>> """
> >>>> general purpose method to save data (value) in the cache
> >>>>
> >>>> :param key (string) key of the value to be saved in cache
> >>>> :param value (any type) the value to be saved
> >>>> """
> >>>> region.set(key, value)
> >>>>
> >>>>
> >>>> def get(key):
> >>>> """
> >>>> general purpose method to get data from the cache
> >>>>
> >>>> :param key (string) key of the data to be fetched
> >>>> :return value (any type) data to be returned from the cache
> >>>> """
> >>>> return region.get(key)
> >>>>
> >>>>
> >>>> [2] My python example
> >>>>
> >>>> `setter.py`
> >>>>
> >>>> def myset(value):
> >>>> Memoize.save("myvalue", value)
> >>>>
> >>>> `getter.py`
> >>>>
> >>>> def myget():
> >>>> return Memoize.get("myvalue") <- this value is NO_VALUE. NO_VALUE
> >>>>
> >>>> My class:
> >>>>
> >>>> setter.myset(123)
> >>>> getter.myget()
> >>>
> >>> The idea that I get from dogpile, is that in each module (getter.py, or
> >>> setter.py) there is a dictionary where the values are stored in the
> >>> backend. Hence, getter.py has its dictionary and setter.py has its
> >>> dictionary also. In the end, there is not a single dictionary where all
> >>> the values should be put. And I want a single dictionary.
> >>>
> >>
> >> Then put everything in one file. Three files for the amount of code you
> >> show above is nonsensical. You might like to read
> >> http://dirtsimple.org/2004/12/python-is-not-java.html and in response to
> >> that http://dirtsimple.org/2004/12/java-is-not-python-either.html
> >>
> >> --
> >> My fellow Pythonistas, ask not what our language can do for you, ask
> >> what you can do for our language.
> >>
> >> Mark Lawrence
> >
> > Here is the full class that I use to store the data.
> >
> > from dogpile.cache import make_region
> >
> >
> > # my_dictionary = {}
> > region = make_region().configure('dogpile.cache.memory')
> > # arguments={"cache_dict":my_dictionary})
> > class Cache:
> >
> > @staticmethod
> > def save(key, value):
> > """
> > general purpose method to save data (value) in the cache
> >
> > :param key (string) key of the value to be saved in cache
> > :param value (any type) the value to be saved
> > """
> > region.set(key, value)
> >
> > @staticmethod
> > def get(key):
> > """
> > general purpose method to get data from the cache
> >
> > :param key (string) key of the data to be fetched
> > :re
Path problems when I am in bash
I have my source code inside the directory `medusa`, and my unit tests inside
`tests` dir. Both dirs are inside `medusa-2.0` dir. Here is my file structure
[1].
When I run my tests inside pycharm, everything works fine, but when I try to
run my unit tests inside in the prompt [2], the
`scheduler/predictionranking.py` can't find the `hdfs` import [3]. I have set
my environment inside the `medusa-2.0` dir with the virtualenv, I even have set
`medusa-2.0` in the PYTHON_PATH [4]. The error that I have is in [5].
The question is, why the import is not being done correctly?
[1] My file structure
medusa-2.0$
medusa (source code)
hdfs.py
scheduler (dir with my schedulers)
predictionranking.py
tests (my unit tests)
[2] I run the unit test like this.
medusa-2.0$ python -v tests/testSimpleRun.py
[3] The header in `predictionranking.py`
import hdfs
def get_prediction_metrics(clusters, pinput):
"""
:param pinput (list) list of input paths
"""
input_size = hdfs.get_total_size(pinput)
[4] My python path
export MEDUSA_HOME=$HOME/repositories/git/medusa-2.0
export PYTHONPATH=${PYTHONPATH}:${MEDUSA_HOME}/medusa
[5] error that I have
medusa-2.0$ python -v tests/testSimpleRun.py
# /home/xeon/repositories/git/medusa-2.0/medusa/local.pyc matches
/home/xeon/repositories/git/medusa-2.0/medusa/local.py
import medusa.local # precompiled from
/home/xeon/repositories/git/medusa-2.0/medusa/local.pyc
# /home/xeon/repositories/git/medusa-2.0/medusa/ranking.pyc matches
/home/xeon/repositories/git/medusa-2.0/medusa/ranking.py
import medusa.ranking # precompiled from
/home/xeon/repositories/git/medusa-2.0/medusa/ranking.pyc
# /home/xeon/repositories/git/medusa-2.0/medusa/decors.pyc matches
/home/xeon/repositories/git/medusa-2.0/medusa/decors.py
import medusa.decors # precompiled from
/home/xeon/repositories/git/medusa-2.0/medusa/decors.pyc
# /home/xeon/repositories/git/medusa-2.0/medusa/settings.pyc matches
/home/xeon/repositories/git/medusa-2.0/medusa/settings.py
import medusa.settings # precompiled from
/home/xeon/repositories/git/medusa-2.0/medusa/settings.pyc
import medusa.scheduler # directory
/home/xeon/repositories/git/medusa-2.0/medusa/scheduler
# /home/xeon/repositories/git/medusa-2.0/medusa/scheduler/__init__.pyc matches
/home/xeon/repositories/git/medusa-2.0/medusa/scheduler/__init__.py
import medusa.scheduler # precompiled from
/home/xeon/repositories/git/medusa-2.0/medusa/scheduler/__init__.pyc
# /home/xeon/repositories/git/medusa-2.0/medusa/scheduler/predictionranking.pyc
matches
/home/xeon/repositories/git/medusa-2.0/medusa/scheduler/predictionranking.py
import medusa.scheduler.predictionranking # precompiled from
/home/xeon/repositories/git/medusa-2.0/medusa/scheduler/predictionranking.pyc
Traceback (most recent call last):
File "tests/simpleRun.py", line 4, in
from medusa.simplealgorithm import run_simple_execution
File "/home/xeon/repositories/git/medusa-2.0/medusa/simplealgorithm.py", line
8, in
import accidentalfaults
File "/home/xeon/repositories/git/medusa-2.0/medusa/accidentalfaults.py",
line 5, in
import hdfs
File "/home/xeon/repositories/git/medusa-2.0/medusa/hdfs.py", line 6, in
from system import execute_command
File "/home/xeon/repositories/git/medusa-2.0/medusa/system.py", line 10, in
from ranking import rank_clusters
File "/home/xeon/repositories/git/medusa-2.0/medusa/ranking.py", line 9, in
from scheduler.predictionranking import get_prediction_metrics
File
"/home/xeon/repositories/git/medusa-2.0/medusa/scheduler/predictionranking.py",
line 6, in
import hdfs
--
https://mail.python.org/mailman/listinfo/python-list
Re: Path problems when I am in bash
On Wednesday, December 30, 2015 at 2:30:40 AM UTC, Karim wrote:
> On 30/12/2015 00:21, xeon Mailinglist wrote:
> > I have my source code inside the directory `medusa`, and my unit tests
> > inside `tests` dir. Both dirs are inside `medusa-2.0` dir. Here is my file
> > structure [1].
> >
> > When I run my tests inside pycharm, everything works fine, but when I try
> > to run my unit tests inside in the prompt [2], the
> > `scheduler/predictionranking.py` can't find the `hdfs` import [3]. I have
> > set my environment inside the `medusa-2.0` dir with the virtualenv, I even
> > have set `medusa-2.0` in the PYTHON_PATH [4]. The error that I have is in
> > [5].
> >
> > The question is, why the import is not being done correctly?
> >
> >
> > [1] My file structure
> >
> > medusa-2.0$
> > medusa (source code)
> > hdfs.py
> > scheduler (dir with my schedulers)
> > predictionranking.py
> > tests (my unit tests)
> >
> >
> > [2] I run the unit test like this.
> >
> > medusa-2.0$ python -v tests/testSimpleRun.py
> >
> >
> > [3] The header in `predictionranking.py`
> >
> > import hdfs
> >
> > def get_prediction_metrics(clusters, pinput):
> > """
> > :param pinput (list) list of input paths
> >
> > """
> >
> > input_size = hdfs.get_total_size(pinput)
> >
> > [4] My python path
> >
> > export MEDUSA_HOME=$HOME/repositories/git/medusa-2.0
> >
> > export PYTHONPATH=${PYTHONPATH}:${MEDUSA_HOME}/medusa
> >
> > [5] error that I have
> >
> > medusa-2.0$ python -v tests/testSimpleRun.py
> > # /home/xeon/repositories/git/medusa-2.0/medusa/local.pyc matches
> > /home/xeon/repositories/git/medusa-2.0/medusa/local.py
> > import medusa.local # precompiled from
> > /home/xeon/repositories/git/medusa-2.0/medusa/local.pyc
> > # /home/xeon/repositories/git/medusa-2.0/medusa/ranking.pyc matches
> > /home/xeon/repositories/git/medusa-2.0/medusa/ranking.py
> > import medusa.ranking # precompiled from
> > /home/xeon/repositories/git/medusa-2.0/medusa/ranking.pyc
> > # /home/xeon/repositories/git/medusa-2.0/medusa/decors.pyc matches
> > /home/xeon/repositories/git/medusa-2.0/medusa/decors.py
> > import medusa.decors # precompiled from
> > /home/xeon/repositories/git/medusa-2.0/medusa/decors.pyc
> > # /home/xeon/repositories/git/medusa-2.0/medusa/settings.pyc matches
> > /home/xeon/repositories/git/medusa-2.0/medusa/settings.py
> > import medusa.settings # precompiled from
> > /home/xeon/repositories/git/medusa-2.0/medusa/settings.pyc
> > import medusa.scheduler # directory
> > /home/xeon/repositories/git/medusa-2.0/medusa/scheduler
> > # /home/xeon/repositories/git/medusa-2.0/medusa/scheduler/__init__.pyc
> > matches /home/xeon/repositories/git/medusa-2.0/medusa/scheduler/__init__.py
> > import medusa.scheduler # precompiled from
> > /home/xeon/repositories/git/medusa-2.0/medusa/scheduler/__init__.pyc
> > #
> > /home/xeon/repositories/git/medusa-2.0/medusa/scheduler/predictionranking.pyc
> > matches
> > /home/xeon/repositories/git/medusa-2.0/medusa/scheduler/predictionranking.py
> > import medusa.scheduler.predictionranking # precompiled from
> > /home/xeon/repositories/git/medusa-2.0/medusa/scheduler/predictionranking.pyc
> > Traceback (most recent call last):
> >File "tests/simpleRun.py", line 4, in
> > from medusa.simplealgorithm import run_simple_execution
> >File "/home/xeon/repositories/git/medusa-2.0/medusa/simplealgorithm.py",
> > line 8, in
> > import accidentalfaults
> >File
> > "/home/xeon/repositories/git/medusa-2.0/medusa/accidentalfaults.py", line
> > 5, in
> > import hdfs
> >File "/home/xeon/repositories/git/medusa-2.0/medusa/hdfs.py", line 6, in
> >
> > from system import execute_command
> >File "/home/xeon/repositories/git/medusa-2.0/medusa/system.py", line 10,
> > in
> > from ranking import rank_clusters
> >File "/home/xeon/repositories/git/medusa-2.0/medusa/ranking.py", line 9,
> > in
> > from scheduler.predictionranking import get_prediction_metrics
> >File
> > "/home/xeon/repositories/git/medusa-2.0/medusa/scheduler/predictionranking.py",
> > line 6, in
> > import hdfs
>
> Hello,
>
> Can you try to set your PYTHONPATH like that:
>
> export PYTHONPATH=${PYTHONPATH}:${MEDUSA_HOME}
>
> Regards
> Karim
Thanks.
--
https://mail.python.org/mailman/listinfo/python-list
process do not join?
I am trying to debug my program that launch processes to run a function to copy
data between hosts located really far away from each other. The end of my
function are in the orders.py and mergedirs.py.
>From this point onwards, in is python code. The problem is that this code
>hangs in the last instruction "forking()", instead of my program continue.
The call of my function were made with: outputs = parmap(mergeDirs, args). This
is my parmap function.
It seems that the function mergeDirs ends, but the process does not join. I
checked that the mergeDirs is working properly, because I ran the code without
processes, and it works fine.
I don't know what is happening. How can I debug this problem?
def parmap(f, X):
pipe = [Pipe() for x in X]
# 2 - what is happening with the tuples (c,x) and (p, c)?
proc = [Process(target=spawn(f), args=(c, x))
for x, (p, c) in zip(X, pipe)]
#[p.start() for p in proc]
for p in proc:
print("Spawn")
p.start()
#[p.join() for p in proc]
for p in proc:
print("Joining")
p.join()
return [p.recv() for (p, c) in pipe]
Copy data time: 104.863273859
orders.py(99): return cluster
mergedirs.py(48): return (new_included, command, poutput)
spawn.py(7): pipe.close()
process.py(259): exitcode = 0
process.py(261): util._exit_function()
--- modulename: util, funcname: _exit_function
util.py(303): info('process shutting down')
--- modulename: util, funcname: info
util.py(77): if _logger:
util.py(304): debug('running all "atexit" finalizers with priority >= 0')
--- modulename: util, funcname: debug
util.py(73): if _logger:
util.py(305): _run_finalizers(0)
--- modulename: util, funcname: _run_finalizers
util.py(257): if _finalizer_registry is None:
util.py(263): if minpriority is None:
util.py(266): f = lambda p : p[0][0] is not None and p[0][0] >=
minpriority
util.py(268): items = [x for x in _finalizer_registry.items() if f(x)]
util.py(269): items.sort(reverse=True)
util.py(271): for key, finalizer in items:
util.py(279): if minpriority is None:
util.py(307): if current_process() is not None:
--- modulename: process, funcname: current_process
process.py(63): return _current_process
util.py(318): for p in active_children():
--- modulename: process, funcname: active_children
process.py(69): _cleanup()
--- modulename: process, funcname: _cleanup
process.py(78): for p in list(_current_process._children):
process.py(70): return list(_current_process._children)
util.py(323): for p in active_children():
--- modulename: process, funcname: active_children
process.py(69): _cleanup()
--- modulename: process, funcname: _cleanup
process.py(78): for p in list(_current_process._children):
process.py(70): return list(_current_process._children)
util.py(327): debug('running the remaining "atexit" finalizers')
--- modulename: util, funcname: debug
util.py(73): if _logger:
util.py(328): _run_finalizers()
--- modulename: util, funcname: _run_finalizers
util.py(257): if _finalizer_registry is None:
util.py(263): if minpriority is None:
util.py(264): f = lambda p : p[0][0] is not None
util.py(268): items = [x for x in _finalizer_registry.items() if f(x)]
util.py(269): items.sort(reverse=True)
util.py(271): for key, finalizer in items:
util.py(279): if minpriority is None:
util.py(280): _finalizer_registry.clear()
process.py(278): util.info('process exiting with exitcode %d' %
exitcode)
--- modulename: util, funcname: info
util.py(77): if _logger:
process.py(279): return exitcode
forking.py(127): sys.stdout.flush()
forking.py(128): sys.stderr.flush()
forking.py(129): os._exit(code)
--
https://mail.python.org/mailman/listinfo/python-list
Re: process do not join?
This log came when I launched the command:
python -m trace --trace myclass.py
On Saturday, April 5, 2014 3:18:34 AM UTC+1, xeon Mailinglist wrote:
> I am trying to debug my program that launch processes to run a function to
> copy data between hosts located really far away from each other. The end of
> my function are in the orders.py and mergedirs.py.
>
>
>
> From this point onwards, in is python code. The problem is that this code
> hangs in the last instruction "forking()", instead of my program continue.
>
>
>
> The call of my function were made with: outputs = parmap(mergeDirs, args).
> This is my parmap function.
>
>
>
> It seems that the function mergeDirs ends, but the process does not join. I
> checked that the mergeDirs is working properly, because I ran the code
> without processes, and it works fine.
>
>
>
> I don't know what is happening. How can I debug this problem?
>
>
>
> def parmap(f, X):
>
> pipe = [Pipe() for x in X]
>
> # 2 - what is happening with the tuples (c,x) and (p, c)?
>
> proc = [Process(target=spawn(f), args=(c, x))
>
> for x, (p, c) in zip(X, pipe)]
>
> #[p.start() for p in proc]
>
> for p in proc:
>
> print("Spawn")
>
> p.start()
>
> #[p.join() for p in proc]
>
> for p in proc:
>
> print("Joining")
>
> p.join()
>
> return [p.recv() for (p, c) in pipe]
>
>
>
>
>
>
>
>
>
> Copy data time: 104.863273859
>
> orders.py(99): return cluster
>
> mergedirs.py(48): return (new_included, command, poutput)
>
> spawn.py(7): pipe.close()
>
> process.py(259): exitcode = 0
>
> process.py(261): util._exit_function()
>
> --- modulename: util, funcname: _exit_function
>
> util.py(303): info('process shutting down')
>
> --- modulename: util, funcname: info
>
> util.py(77): if _logger:
>
> util.py(304): debug('running all "atexit" finalizers with priority >= 0')
>
> --- modulename: util, funcname: debug
>
> util.py(73): if _logger:
>
> util.py(305): _run_finalizers(0)
>
> --- modulename: util, funcname: _run_finalizers
>
> util.py(257): if _finalizer_registry is None:
>
> util.py(263): if minpriority is None:
>
> util.py(266): f = lambda p : p[0][0] is not None and p[0][0] >=
> minpriority
>
> util.py(268): items = [x for x in _finalizer_registry.items() if f(x)]
>
> util.py(269): items.sort(reverse=True)
>
> util.py(271): for key, finalizer in items:
>
> util.py(279): if minpriority is None:
>
> util.py(307): if current_process() is not None:
>
> --- modulename: process, funcname: current_process
>
> process.py(63): return _current_process
>
> util.py(318): for p in active_children():
>
> --- modulename: process, funcname: active_children
>
> process.py(69): _cleanup()
>
> --- modulename: process, funcname: _cleanup
>
> process.py(78): for p in list(_current_process._children):
>
> process.py(70): return list(_current_process._children)
>
> util.py(323): for p in active_children():
>
> --- modulename: process, funcname: active_children
>
> process.py(69): _cleanup()
>
> --- modulename: process, funcname: _cleanup
>
> process.py(78): for p in list(_current_process._children):
>
> process.py(70): return list(_current_process._children)
>
> util.py(327): debug('running the remaining "atexit" finalizers')
>
> --- modulename: util, funcname: debug
>
> util.py(73): if _logger:
>
> util.py(328): _run_finalizers()
>
> --- modulename: util, funcname: _run_finalizers
>
> util.py(257): if _finalizer_registry is None:
>
> util.py(263): if minpriority is None:
>
> util.py(264): f = lambda p : p[0][0] is not None
>
> util.py(268): items = [x for x in _finalizer_registry.items() if f(x)]
>
> util.py(269): items.sort(reverse=True)
>
> util.py(271): for key, finalizer in items:
>
> util.py(279): if minpriority is None:
>
> util.py(280): _finalizer_registry.clear()
>
> process.py(278): util.info('process exiting with exitcode %d' %
> exitcode)
>
> --- modulename: util, funcname: info
>
> util.py(77): if _logger:
>
> process.py(279): return exitcode
>
> forking.py(127): sys.stdout.flush()
>
> forking.py(128): sys.stderr.flush()
>
> forking.py(129): os._exit(code)
--
https://mail.python.org/mailman/listinfo/python-list
celery errors in eclipse
I am using celery in my program to invoke remote functions using a message queueing service (RabbitMQ). I call my function with celery with this: myfunc.apply_async(queue=cluster, args=(chain, cluster, )), but although the call is correct and it works, in eclipse I get the error that "undefined variable from import: apply_async". This happens, because to celery methods it is just need to append celery methods to my function, and not code them. That is true that I didn`t define apply_async function, nor imported. I just use it like celery docs suggests. Eclipse can`t see this. The celery community say that it is eclipse python linter causing the issue and they suggest to contact the pydev community regarding the matter. How I make this error disappears in the eclipse? -- https://mail.python.org/mailman/listinfo/python-list
