Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
On Thu, 3 Oct 2024 at 08:48, Left Right wrote: > > > You can't validate an IP packet without having all of it. Your notion > > of "streaming" is nonsensical. > > Whoa, whoa, hold your horses! "nonsensical" needs a little bit of > justification :) > > It seems you don't understand the difference between words and > languages! In my examples, IP _protocol_ is the language, sequences of > IP packets are the words in the language. A language is amenable to > streaming if the words of the language are repetition of sequences of > symbols of the alphabet of fixed length. This is, essentially, like > saying that the words themselves are regular. One single IP packet is all you can parse. You're playing shenanigans with words the way Humpty Dumpty does. IP packets are not sequences, they are individuals. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Python crash together with threads
My Software project is working fine in most of the cases
(www.pythonscad.org)
however I am right now isolating a scenario, which makes it crash
permanently.
It does not happen with Python 3.11.6 (and possibly below), it happens with
3.12 and above
It does not happen when not using Threads.
However due to the architecture of the program I am forced to evaluate some
parts in main thread and some parts in a dedicated Thread. The Thread is
started with QThread(QT 5.0)
whereas I am quite sure that program flows do not overlap.
When I just execute my 1st very simple Python function inside the newly
created thread, like:
PyObject *a = PyFloat_FromDouble(3.3);
my program crashes with this Stack trace
0 0x7f6837fe000f in _PyInterpreterState_GET () at
./Include/internal/pycore_pystate.h:179
#1 get_float_state () at Objects/floatobject.c:38
#2 PyFloat_FromDouble (fval=3.2998) at
Objects/floatobject.c:136
#3 0x015a021f in python_testfunc() ()
#4 0x01433301 in CGALWorker::work() ()
#5 0x00457135 in CGALWorker::qt_static_metacall(QObject*,
QMetaObject::Call, int, void**) ()
#6 0x7f68364d0f9f in void doActivate(QObject*, int, void**) ()
at /lib64/libQt5Core.so.5
#7 0x7f68362e66ee in QThread::started(QThread::QPrivateSignal) () at
/lib64/libQt5Core.so.5
#8 0x7f68362e89c4 in QThreadPrivate::start(void*) () at
/lib64/libQt5Core.so.5
#9 0x7f6835cae19d in start_thread () at /lib64/libc.so.6
#10 0x7f6835d2fc60 in clone3 () at /lib64/libc.so.6
I suspect, that this is a Null pointer here
See also _PyInterpreterState_Get()
and _PyGILState_GetInterpreterStateUnsafe(). */
static inline PyInterpreterState* _PyInterpreterState_GET(void) {
PyThreadState *tstate = _PyThreadState_GET();
#ifdef Py_DEBUG
_Py_EnsureTstateNotNULL(tstate);
#endif
# <<--- suspect state is nullpointer
return tstate->interp;
}
any clues , whats going on here, and how I can mitigate that ?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
> One single IP packet is all you can parse. I worked for an undisclosed company which manufactures h/w for ISPs (4- and 8-unit boxes you mount on a rack in a datacenter). Essentially, big-big routers. So, I had the pleasure of writing software that parses IP _protocol_, and let me tell you: you have no idea what you just wrote. But, like I wrote earlier: you don't understand the distinction between languages and words. And in general, are just being stubborn and rude because you are trying to prove a point to someone you don't like, but, in reality, you just look more and more ridiculous. On Thu, Oct 3, 2024 at 12:51 AM Chris Angelico wrote: > > On Thu, 3 Oct 2024 at 08:48, Left Right wrote: > > > > > You can't validate an IP packet without having all of it. Your notion > > > of "streaming" is nonsensical. > > > > Whoa, whoa, hold your horses! "nonsensical" needs a little bit of > > justification :) > > > > It seems you don't understand the difference between words and > > languages! In my examples, IP _protocol_ is the language, sequences of > > IP packets are the words in the language. A language is amenable to > > streaming if the words of the language are repetition of sequences of > > symbols of the alphabet of fixed length. This is, essentially, like > > saying that the words themselves are regular. > > One single IP packet is all you can parse. You're playing shenanigans > with words the way Humpty Dumpty does. IP packets are not sequences, > they are individuals. > > ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python crash together with threads
On 10/2/2024 7:26 AM, Guenther Sohler wrote:
My Software project is working fine in most of the cases
(www.pythonscad.org)
however I am right now isolating a scenario, which makes it crash
permanently.
It does not happen with Python 3.11.6 (and possibly below), it happens with
3.12 and above
It does not happen when not using Threads.
However due to the architecture of the program I am forced to evaluate some
parts in main thread and some parts in a dedicated Thread. The Thread is
started with QThread(QT 5.0)
whereas I am quite sure that program flows do not overlap.
When I just execute my 1st very simple Python function inside the newly
created thread, like:
PyObject *a = PyFloat_FromDouble(3.3);
my program crashes with this Stack trace
0 0x7f6837fe000f in _PyInterpreterState_GET () at
./Include/internal/pycore_pystate.h:179
#1 get_float_state () at Objects/floatobject.c:38
#2 PyFloat_FromDouble (fval=3.2998) at
Objects/floatobject.c:136
#3 0x015a021f in python_testfunc() ()
#4 0x01433301 in CGALWorker::work() ()
#5 0x00457135 in CGALWorker::qt_static_metacall(QObject*,
QMetaObject::Call, int, void**) ()
#6 0x7f68364d0f9f in void doActivate(QObject*, int, void**) ()
at /lib64/libQt5Core.so.5
#7 0x7f68362e66ee in QThread::started(QThread::QPrivateSignal) () at
/lib64/libQt5Core.so.5
#8 0x7f68362e89c4 in QThreadPrivate::start(void*) () at
/lib64/libQt5Core.so.5
#9 0x7f6835cae19d in start_thread () at /lib64/libc.so.6
#10 0x7f6835d2fc60 in clone3 () at /lib64/libc.so.6
I suspect, that this is a Null pointer here
See also _PyInterpreterState_Get()
and _PyGILState_GetInterpreterStateUnsafe(). */
static inline PyInterpreterState* _PyInterpreterState_GET(void) {
PyThreadState *tstate = _PyThreadState_GET();
#ifdef Py_DEBUG
_Py_EnsureTstateNotNULL(tstate);
#endif
# <<--- suspect state is nullpointer
return tstate->interp;
}
any clues , whats going on here, and how I can mitigate that ?
Can you post a small, self-contained program that demonstrates the problem?
Louis
--
https://mail.python.org/mailman/listinfo/python-list
Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
> You can't validate an IP packet without having all of it. Your notion > of "streaming" is nonsensical. Whoa, whoa, hold your horses! "nonsensical" needs a little bit of justification :) It seems you don't understand the difference between words and languages! In my examples, IP _protocol_ is the language, sequences of IP packets are the words in the language. A language is amenable to streaming if the words of the language are repetition of sequences of symbols of the alphabet of fixed length. This is, essentially, like saying that the words themselves are regular. So, the follow-up question from you to me should be: how come strictly context-free languages can still be parsed with streaming parsers? -- And the answer to that is it's possible to approximate context-free languages with regular languages. In fact, this is a very interesting subject, which unfortunately is usually overlooked in automata classes. It's interesting in a sense that it's very accessible to the students who already mastered the understanding of regular and context-free formalisms. So, streaming parsers (eg. SAX) are written for a regular language that approximates XML. This is because in practice we will almost never encounter more than N nesting levels in an XML, more than N characters in an element name etc. (for some large enough N). Something which allows us to create a regular language from a context-free one. NB. "Nonsensical" has a very precise meaning, when it comes to discussing the truth value of a proposition, which I think you also somehow didn't know about. You seem to use "nonsensical" as a synonym to "wrong". But, unbeknownst to you, you said something else. You actually implied that there's no way to tell if my notion of streaming is correct or not. But, for the future reference: my notion of streaming is correct, and you would do better learning some materials about it before jumping to conclusions. -- https://mail.python.org/mailman/listinfo/python-list
Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
> By that definition of "streaming", no parser can ever be streaming, > because there will be some constructs that must be read in their > entirety before a suitably-structured piece of output can be > emitted. In the same email you replied to, I gave examples of languages for which parsers can be streaming (in general): SCSI or IP. For some languages (eg. everything in the context-free family) streaming parsers are _in general_ impossible, because there are pathological cases like the one with parsing numbers. But this doesn't mean that you cannot come up with a parser that is only useful _sometimes_. And, in practice, languages like XML or JSON do well with streaming, even though in general it's impossible. I'm sorry if this comes as a surprise. On one hand I don't want to sound condescending, on the other hand, this is something that you'd typically study in automata theory class. Well, not exactly in the very same words, but you should be able to figure this stuff out if you had that class. -- https://mail.python.org/mailman/listinfo/python-list
Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
On Wed, 2 Oct 2024 at 23:53, Left Right via Python-list wrote: > In the same email you replied to, I gave examples of languages for > which parsers can be streaming (in general): SCSI or IP. You can't validate an IP packet without having all of it. Your notion of "streaming" is nonsensical. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
This thread is derailing. Please consider it closed. -- ~Ethan~ Moderator -- https://mail.python.org/mailman/listinfo/python-list
