Hi Joe,
That stack trace isn't very helpful because:
1. The exception description is missing.
2. The source file names are missing (despite line numbers being present).
3. It's a synchronous trace from the point where the exception was thrown,
so it mostly just shows the event loop machinery rather than tracing
through application code.
If you log the exception at the point where it is caught, you should get a
more useful stack trace that traces through the asynchronous call stack,
provided by KJ. What I'm trying to figure out is what high-level API you
called that threw the exception.
FWIW you can distinguish whether an exception is caused by a network
disconnect by examining its type, i.e.:
if (exception.getType() == kj::Exception::DISCONNECTED)
That said, normally in a shutdown scenario I would delete all the promises
in order to proactively cancel the tasks they represent, rather than wait
for them to fail with DISCONNECTED. The KJ promise framework is designed to
allow for cancellation in this way.
(Also if the process is exiting, then I'd just call exit() and be done with
it. "Clean teardown" before process exit is usually a waste of time.)
-Kenton
On Wed, Jul 3, 2019 at 8:43 AM Joe Ludwig <[email protected]> wrote:
> Hi,
>
> I'm hitting a variety of exceptions on shutdown. Exactly what I hit seems
> to shift as I move things around to try to solve the problem, but here's
> the current one. This doesn't happen 100% of the time, so there must be a
> race involved:
>
>> > avrenderer.exe!aardvark::CServerThread::Run::__l4::<lambda>(kj::Exception
>> && exception) Line 683 C++
>> [External Code]
>>
>> avrenderer.exe!aardvark::CRecoverableExceptionHandler::onRecoverableException(kj::Exception
>> && exception) Line 656 C++
>> avrenderer.exe!kj::throwRecoverableException(kj::Exception &&
>> exception, unsigned int ignoreCount) Line 878 C++
>> avrenderer.exe!kj::_::Debug::Fault::~Fault() Line 332 C++
>>
>> avrenderer.exe!kj::`anonymous-namespace'::AsyncStreamFd::read::__l2::<lambda>(unsigned
>> __int64 result) Line 242 C++
>> avrenderer.exe!kj::_::MaybeVoidCaller<unsigned __int64,unsigned
>> __int64>::apply<unsigned __int64 <lambda>(unsigned __int64)
>> >(kj::`anonymous-namespace'::AsyncStreamFd::read::__l2::unsigned __int64
>> <lambda>(unsigned __int64) & func, unsigned __int64 && in) Line 135 C++
>> avrenderer.exe!kj::_::TransformPromiseNode<unsigned __int64,unsigned
>> __int64,unsigned __int64 <lambda>(unsigned
>> __int64),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue &
>> output) Line 401 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>()
>> Line 703 C++
>> avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line
>> 303 C++
>> avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable &
>> runnable) Line 1023 C++
>> avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void)
>> >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func)
>> Line 315 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue
>> & output) Line 703 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue
>> & output) Line 721 C++
>> avrenderer.exe!kj::_::TransformPromiseNode<kj::_::Void,unsigned
>> __int64,void <lambda>(unsigned
>> __int64),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue &
>> output) Line 396 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>()
>> Line 703 C++
>> avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line
>> 303 C++
>> avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable &
>> runnable) Line 1023 C++
>> avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void)
>> >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func)
>> Line 315 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue
>> & output) Line 703 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue
>> & output) Line 721 C++
>> avrenderer.exe!kj::_::TransformPromiseNode<bool,kj::_::Void,bool
>> <lambda>(void),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue
>> & output) Line 396 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>()
>> Line 703 C++
>> avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line
>> 303 C++
>> avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable &
>> runnable) Line 1023 C++
>> avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void)
>> >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func)
>> Line 315 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue
>> & output) Line 703 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue
>> & output) Line 721 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNode<kj::Maybe<kj::Own<capnp::MessageReader>
>> >,bool,kj::CaptureByMove<kj::Maybe<kj::Own<capnp::MessageReader> >
>> <lambda>(kj::Own<capnp::MessageReader> &&, bool),kj::Own<capnp::`anonymous
>> namespace'::AsyncMessageReader>
>> >,kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output)
>> Line 396 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>()
>> Line 703 C++
>> avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line
>> 303 C++
>> avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable &
>> runnable) Line 1023 C++
>> avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void)
>> >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func)
>> Line 315 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue
>> & output) Line 703 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue
>> & output) Line 721 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNode<kj::Maybe<kj::Own<capnp::IncomingRpcMessage>
>> >,kj::Maybe<kj::Own<capnp::MessageReader>
>> >,kj::Maybe<kj::Own<capnp::IncomingRpcMessage> >
>> <lambda>(kj::Maybe<kj::Own<capnp::MessageReader> >
>> &&),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output)
>> Line 396 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>()
>> Line 703 C++
>> avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line
>> 303 C++
>> avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable &
>> runnable) Line 1023 C++
>> avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void)
>> >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func)
>> Line 315 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue
>> & output) Line 703 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue
>> & output) Line 721 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNode<bool,kj::Maybe<kj::Own<capnp::IncomingRpcMessage>
>> >,bool <lambda>(kj::Maybe<kj::Own<capnp::IncomingRpcMessage> >
>> &&),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output)
>> Line 396 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>()
>> Line 703 C++
>> avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line
>> 303 C++
>> avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable &
>> runnable) Line 1023 C++
>> avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void)
>> >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func)
>> Line 315 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue
>> & output) Line 703 C++
>>
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue
>> & output) Line 721 C++
>> avrenderer.exe!kj::_::TransformPromiseNode<kj::_::Void,bool,void
>> <lambda>(bool),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue
>> & output) Line 396 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>()
>> Line 703 C++
>> avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line
>> 303 C++
>> avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable &
>> runnable) Line 1023 C++
>> avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void)
>> >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func)
>> Line 315 C++
>> avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue
>> & output) Line 703 C++
>> avrenderer.exe!kj::TaskSet::Task::fire() Line 180 C++
>> avrenderer.exe!kj::EventLoop::turn() Line 373 C++
>> avrenderer.exe!kj::WaitScope::poll() Line 413 C++
>> avrenderer.exe!aardvark::CServerThread::Run() Line 689 C++
>
>
> The specific exception in this case is that AsyncStreamFd wants to read
> some bytes but none are available, presumably because the client has
> disconnected.
>
> The overall flow of the system is something like this:
>
> 1. server starts up
> 2. several clients start up. Some are in-process. Some are other
> processes on the same system.
> 3. Run for a while
> 4. Clean up all the clients
> 5. sleep(1000) <-- this is where the exceptions happen
> 6. clean up the server
>
> The sleep is only there to help me sort out this problem. Once I make
> shutdown deterministic it'll go away. But it appears that clients
> disconnecting and being freed is actually causing an exception on the
> server.
>
> Is there some way for a server to handle client disconnections at
> arbitrary times without throwing? I could just swallow all exceptions as
> "must be a disconnect", but that doesn't seem quite right either since
> there could also be bugs in my own server code causing exceptions. I could
> make changes client-side too, but since clients can always just crash and
> not tell me about it that doesn't fully solve things.
>
> I *am* able to disconnect and reconnect clients while running in general.
> This seems to be unique to shutdown in some way I don't understand. (The
> server doesn't know it's shutting down yet when the exception occurs.)
>
> Any ideas?
>
>
> Joe
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> Visit this group at https://groups.google.com/group/capnproto.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/b816998f-a402-4f7f-a7ff-c2ace394f3b7%40googlegroups.com
> <https://groups.google.com/d/msgid/capnproto/b816998f-a402-4f7f-a7ff-c2ace394f3b7%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
Visit this group at https://groups.google.com/group/capnproto.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/CAJouXQmj-TG47G_M5Ep7ZLaUbdJRL3P1j5wYRP7zJSEEm6t6sQ%40mail.gmail.com.
