I've got a strange problem with an app running on NodeJS, which I'd
appreciate any help with.
The problem manifests as either TypeError trying to access objects that
haven't been initialized consistently, or AssertionError due to asserts in
my code finding violations of invariants that should hold -- in both cases,
it looks like the code was invoked with other data structures in an
inconsistent state. This could just be a bug in my code, sure, except for a
peculiarity with _tickCallback getting called seemingly out of nowhere as
you'll see below.
My understanding of the NodeJS concurrency model is that what's going on
here should be impossible -- I'd explain the concurrency model as, there's
only one thread and no preemption, scheduling happens only at the level of
the event loop, synchronous code runs until it decides to yield by
returning to the event loop, so the only boundary where code needs to worry
about data structures changing out from under it is when scheduling a
callback asynchronously (with a trip back to the event loop dispatch
mechanism in between). Or said more shortly, every synchronous invocation
is essentially a critical section until it returns all the way back to
Node's event loop.
But then I get an exception with a call stack like:
TypeError: Cannot call method 'end' of undefined
at Request.end
(/opt/buildbot-slave/tree/build/node_modules/request/request.js:1320:12)
at
/opt/buildbot-slave/tree/build/node_modules/request/request.js:418:14
at process._tickCallback (node.js:415:13)
at
/opt/buildbot-slave/tree/build/node_modules/request/lib/copy.js:5:10
at Array.forEach (native)
at copy
(/opt/buildbot-slave/tree/build/node_modules/request/lib/copy.js:4:20)
at Request.start
(/opt/buildbot-slave/tree/build/node_modules/request/request.js:668:20)
at Request.end
(/opt/buildbot-slave/tree/build/node_modules/request/request.js:1319:28)
at Object.exports.request
(/opt/buildbot-slave/tree/build/node_modules/soap/lib/http.js:55:11)
Note the call to process._tickCallback in the middle of this stack trace, 3
frames down.
Is this actually legal and possible? According to my understanding of the
NodeJS concurrency model (see above), no; if _tickCallback can be called
from arbitrary points (and then dispatch into other arbitrary code), it
seems like this would break the world. But I haven't found any official
documentation that explicitly rules this out.
I see 3 possibilities:
- this is happening and is legal: then I need to completely rethink my
understanding of data structure consistency in NodeJS.
- this is happening and is not legal: through what bug is it happening,
then?
- it's not actually happening and the call stack is a lie: but then I need
some other explanation for the data structure inconsistency that my
assertions are observing.
Clearly, none of these possibilities is attractive :)
For what it's worth, the code at node_modules/request/lib/copy.js:5:10
which is alleged to have triggered a call to _tickCallback is the second
line of
Object.keys(obj).forEach(function (i) {
o[i] = obj[i]
that is, just a property lookup against a JS object.
I did dig into the NodeJS source trying to find all callers of
_tickCallback; it seems mostly it's used by the C++ code in MakeCallback()
which calls it every time after invoking the JS code "callback".
MakeCallback has a ton of callers, both direct and via a mechanism where
it's registered with libuv as uv_check_immediate callback; I think I read
somewhere that MakeCallback is how native code always invokes JS code, and
so nextTick callbacks are processed (by running process._tickCallback) on
every transition from JS code back to native code.
So my question: is it possible for _tickCallback to be called (and then
dispatch into arbitrary JS code, whatever was registered as a nextTick
callback) from arbitrary JS code, and under what conditions? If it's true
that we invoke _tickCallback via MakeCallback() when leaving JS code back
to native code, does this happen only when the entire synchronous execution
(driven by the last async event) is done and we're going back to sleep in
the event loop, or also if the JS code calls some C++ code in a native-code
extension which itself calls back into JS code (so then unwinding, we go
from JS to C++ but not back to the event loop; we have more JS to run still
on the stack as we unwind; does this scenario also invoke MakeCallback and
possibly _tickCallback)?
Or am I overthinking all this; my initial hunch that _tickCallback should
never show up in the middle of a call stack was true, and this is just a
bug that shouldn't happen?
I'd appreciate knowing where to concentrate my search to understand this
better.
I'm running on Node stable releases (0.10.x; I think I've seen this
behavior since 0.10.16 and it continues in 0.10.26; I haven't tried newer
versions yet). I am using the node-fibers module [1] but I don't think it's
implicated here; fibers have to choose to yield, and if they do yield then
either we go back to the event loop or another fiber starts running from
where it left off; any such fiber interactions would be visible in the call
stack; it should be easy to distinguish between these cases and the one I'm
asking about here. Clearly, that line of code quoted above from
request/lib/copy.js, "o[i] = obj[i]", should not be triggering a fiber
yield, and if it did (say in fancy situations with proxies or getters are
involved), that should not manifest as a call to _tickCallback.
thanks,
Matt
[1]: https://github.com/laverdet/node-fibers
--
Job board: http://jobs.nodejs.org/
New group rules:
https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/nodejs/af1b6c78-ba23-4ec3-ad98-b0b6c148d435%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
