Agreed that all startup crashes are important. The reason I focused on the MOZ_CRASHes is that this is where somebody explicitly said “this is as bad as it gets, we must crash now”, and I’d like to look at those once in a while, and see if that’s really the case. Because I certainly ran into code that was more of the “this should never happen, and if it does I don’t understand why” with MOZ_CRASHes in it, and that can be improved... — - Milan
> On May 31, 2016, at 12:28 , Benjamin Smedberg <benja...@smedbergs.us> wrote: > > You're assuming that this happens every time, instead of randomly. If you add > the time since last crash to your column list, you can see that this is true > in some cases and not others. > > I changed your link a little: > > * remove "moz crash reason exists" - any startup crash is a problem > * excluded content and plugin process crashes - those aren't "startup" > crashes the same way, and they don't prevent users from updating or disabling > addons > * added facets on the crash reason and the abort message > * added the time-since-last-crash to the column list > > The top things on this list are: > > by signature: > > __RtlUserThreadStart | _RtlUserThreadStart (maybe bug 1164826 > <https://bugzilla.mozilla.org/show_bug.cgi?id=1164826> , need to dig to > separate by this DLL) > mozalloc_abort | NS_DebugBreak | ErrorLoadingBuiltinSheet (longstanding > problem, perhaps evidence of a bad update or install - bug 1194856 > <https://bugzilla.mozilla.org/show_bug.cgi?id=1194856>) > OOM | small - this is surprising and disturbing to me - I wonder how much of > this is actually OOM and how much is memory corruption. I filed bug 1276993 > as a work item for this. > EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER - again > normally OOM but I wouldn't expect that at startup > > > --BDS > > > On Tue, May 31, 2016 at 11:52 AM, Milan Sreckovic <msrecko...@mozilla.com > <mailto:msrecko...@mozilla.com>> wrote: > By the way, this is the kiss of death query. MOZ_CRASH, start up, in safe > mode. We’re basically forcing these people away. There is nothing they can > do even if they really want to run Firefox (assuming this is a persistent > start up crash, of course.) The numbers aren’t high, and majority of them > are OOMs, but I still feel like this query should never have things in it. > (Randomly picked 8 seconds, I know that some consider it a start up crash if > it crashes much later than that.) > > https://crash-stats.mozilla.com/search/?product=Firefox&moz_crash_reason=%21__null__&uptime=%3C8&safe_mode=__true__&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature > > <https://crash-stats.mozilla.com/search/?product=Firefox&moz_crash_reason=%21__null__&uptime=%3C8&safe_mode=__true__&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature> > > It’s also interesting to extend the query to specify the amount of memory the > machine has - how do we get an OOM on startup when the users have 2GB of RAM? > — > - Milan > > > > > > >> On May 31, 2016, at 2:22 , Nicholas Nethercote <n.netherc...@gmail.com > >> <mailto:n.netherc...@gmail.com>> wrote: > >> > >> Hi, > >> > >> Here is a crash-stats search that shows all the crash reports in the > >> past 7 days that have a "MozCrashReason" field -- which means they > >> were triggered by MOZ_CRASH or MOZ_RELEASE_ASSERT -- faceted (i.e. > >> aggregated) by that field: > >> > >> https://crash-stats.mozilla.com/search/?product=Firefox&_facets=moz_crash_reason&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=moz_crash_reason#facet-moz_crash_reason > >> > >> <https://crash-stats.mozilla.com/search/?product=Firefox&_facets=moz_crash_reason&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=moz_crash_reason#facet-moz_crash_reason> > >> > >> I've included the output below. (Apologies if it gets munged when the > >> email is processed; just click on the link above to see the live > >> list.) > >> > >> #1 by a long way is shutdown hangs. No great surprise. > >> > >> #2 is unannotated MOZ_CRASH() calls, i.e. there is no string argument > >> given. These are mostly OOMs, though there are a few others in there. > >> These ones should be annotated so they show up separately. > >> > >> From #3 down we have smaller numbers, but many of them are still > >> non-trivial, and a lot of them are probably indicative of problems > >> that are very easy to fix if the right person sees them. Please take a > >> look through the list to see if any of them are familiar to you. > >> > >> (If you're wondering why I made this search... I've found that many > >> crash reports lack enough data to be actionable -- especially those > >> involving crashes caused by bad memory accesses. So it's worth > >> focusing to some degree on the crash reports that are more likely to > >> be actionable, and places where we deliberately abort are an obvious > >> case.) > >> > >> Nick > >> > >> > >> 1 MOZ_CRASH(Shutdown too long, probably frozen, causing a crash.) > >> 129715 9.92 % > >> 2 MOZ_CRASH() 25987 1.99 % > >> 3 MOZ_CRASH(GFX: Unable to get a working D3D9 Compositor) > >> 2104 0.16 % > >> 4 MOZ_CRASH(Unexpected error during FakeBlack creation.) 1679 0.13 % > >> 5 MOZ_CRASH(IPC FatalError in the parent process!) 783 0.06 % > >> 6 MOZ_RELEASE_ASSERT(pi->mInternalRefs < pi->mRefCount) (Cycle > >> collector found more references to an object than its refcount) 509 > >> 0.04 % > >> 7 MOZ_RELEASE_ASSERT(!mDoingStableStates) 466 0.04 % > >> 8 MOZ_CRASH(Bogus tree op) 459 0.04 % > >> 9 MOZ_RELEASE_ASSERT(sAliveDisplayItemDatas && > >> sAliveDisplayItemDatas->Contains(aData)) 263 0.02 % > >> 10 MOZ_CRASH(Using observer service off the main thread!) 223 0.02 % > >> 11 MOZ_RELEASE_ASSERT(!mSkipRequest.Exists()) (called mid-skipping) > >> 222 0.02 % > >> 12 MOZ_CRASH(GFX_CRASH) 215 0.02 % > >> 13 MOZ_RELEASE_ASSERT(NS_IsMainThread()) 131 0.01 % > >> 14 MOZ_RELEASE_ASSERT(aMsg.priority() == > >> IPC::Message::PRIORITY_NORMAL) 120 0.01 % > >> 15 MOZ_RELEASE_ASSERT(aRefCount != 0) (CCed refcounted object has zero > >> refcount) 113 0.01 % > >> 16 MOZ_RELEASE_ASSERT(!mAudio.HasPromise()) (No duplicate sample > >> requests) 110 0.01 % > >> 17 MOZ_RELEASE_ASSERT(ok) 105 0.01 % > >> 18 MOZ_CRASH(GFX: D3D11 timeout) 99 0.01 % > >> 19 MOZ_CRASH(invalid process aSelector) 73 0.01 % > >> 20 MOZ_CRASH(We lost the following char message) 58 0.00 % > >> 21 MOZ_CRASH(Unhandlable OOM while clearing document dependent slots.) > >> 53 0.00 % > >> 22 MOZ_CRASH(TODO: sourcebuffer was deleted from under us) 52 > >> 0.00 % > >> 23 MOZ_RELEASE_ASSERT(prio == IPC::Message::PRIORITY_NORMAL || > >> NS_IsMainThread()) 51 0.00 % > >> 24 MOZ_CRASH(GFX: Failed to update reference draw target after device > >> reset) 45 0.00 % > >> 25 MOZ_RELEASE_ASSERT(isSystem) 42 0.00 % > >> 26 MOZ_CRASH(Crash creating texture. See bug 1221348.) 41 0.00 % > >> 27 MOZ_CRASH(sandbox_init() failed) 41 0.00 % > >> 28 MOZ_CRASH(Unable to get a working D3D9 Compositor) 37 0.00 % > >> 29 MOZ_CRASH(GFX: Invalid D3D11 content device) 33 0.00 % > >> 30 MOZ_CRASH(Initial length is too large) 30 0.00 % > >> 31 MOZ_CRASH(Could not start cubeb stream for MSG.) 27 0.00 % > >> 32 MOZ_CRASH(IPC message size is too large) 25 0.00 % > >> 33 MOZ_RELEASE_ASSERT(mWorkerLoopID == MessageLoop::current()->id()) > >> (not on worker thread!) 25 0.00 % > >> 34 MOZ_RELEASE_ASSERT(!mInWriteTransaction) 24 0.00 % > >> 35 MOZ_CRASH(NativeKey tries to dispatch a key event on destroyed > >> widget) 23 0.00 % > >> 36 MOZ_RELEASE_ASSERT(((bool)(!!(!NS_FAILED_impl(rv)))) && thread) > >> (Should successfully create image decoding threads) 23 0.00 % > >> 37 MOZ_RELEASE_ASSERT(aInAndOutListener) (can not perform CORS checks > >> without a listener) 23 0.00 % > >> 38 MOZ_CRASH(Unknown unit type?) 18 0.00 % > >> 39 MOZ_RELEASE_ASSERT(!mVideo.mDecodingRequested) (Reset must have > >> been called) 17 0.00 % > >> 40 MOZ_RELEASE_ASSERT(msg->size() < IPC::Channel::kMaximumMessageSize) > >> 17 0.00 % > >> 41 MOZ_RELEASE_ASSERT(!r.IsEmpty()) 14 0.00 % > >> 42 MOZ_RELEASE_ASSERT(!r->IsEmpty()) 13 0.00 % > >> 43 MOZ_RELEASE_ASSERT(CheckDocTree()) 12 0.00 % > >> 44 MOZ_RELEASE_ASSERT(mDestroyCalled) 11 0.00 % > >> 45 MOZ_RELEASE_ASSERT(!!compositor) 10 0.00 % > >> 46 MOZ_RELEASE_ASSERT(MessageLoop::current() == mWorkerLoop) 10 > >> 0.00 % > >> 47 MOZ_RELEASE_ASSERT(sDebugOwningThread != currentThread) 10 > >> 0.00 % > >> 48 MOZ_RELEASE_ASSERT(SizeOfEntryStore(CapacityFromHashShift(), > >> mEntrySize, &nbytes)) 8 0.00 % > >> 49 MOZ_CRASH(Accessing the Subject Principal without an AutoJSAPI on > >> the stack is forbidden) 7 0.00 % > >> 50 MOZ_CRASH(Initial entry store size is too large) 7 0.00 % > >> _______________________________________________ > >> dev-platform mailing list > >> dev-platform@lists.mozilla.org <mailto:dev-platform@lists.mozilla.org> > >> https://lists.mozilla.org/listinfo/dev-platform > >> <https://lists.mozilla.org/listinfo/dev-platform> > > > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org <mailto:dev-platform@lists.mozilla.org> > https://lists.mozilla.org/listinfo/dev-platform > <https://lists.mozilla.org/listinfo/dev-platform> > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
