Symbolicating Your iOS Crash Reports

Written by: on March 24, 2015

Updated April 27, 2016

Getting Started

You’ve been handed a crash report for your app but the stack backtrace contains indecipherable memory addresses. What’s a developer to do? In short, you’ll need to apply debugging symbols to the stacktrace to make it human-readable, a process which we call symbolication.

But before we get started, you can follow along using Crasher which provides a sample crash report for you to decipher.

You should have the .crash file. If not, you can grab it from iTunes Connect, directly from a connected device through Xcode (Window > Devices), on a connected device (Settings > Privacy > Diagnostics & Usage), or getting your hands dirty using the PLCrashReporter framework. You might already be using a 3rd party crash report service, which will symbolicate the crash for you when set up correctly.

Depending on how your app’s build was configured, you need one or both of the following:

  • The .app for the application build that crashed. This package contains the app binary, and may contain debug symbols. (If you have an .ipa you can extract the .app from it by uncompressing the file as if it were a .zip.)
  • The .dSYM for the application build that crashed. This is a by-product of your app that contains debug symbols if your .app does not.

Which will you need? In Xcode, look in Build Settings for “Strip Debug Symbols During Copy” (COPY_PHASE_STRIP). When enabled, debug symbols are omitted from your .app and placed into a .dSYM file. Otherwise your .app contains these symbols. (By default, debug symbols are stripped from release builds for reasons of obfuscation. You probably shouldn’t change this setting for the release configuration.)

But Hold On, What’s a Debug Symbol?

For our purposes a debug symbol is the human-readable name a programmer has given to a method. The compiler obfuscates the code by reducing these named debug symbols to its own symbols. You cannot rely on these symbols being the same between builds even if you built identical code twice.

Inspecting the Crash

If you pull the crash report from the device via Xcode’s Organizer, know that your crash report may be symbolicated automatically for UIKit and other iOS frameworks. If Xcode still knows of your build, it will automatically symbolicate your crash.

If this is not the case, you’ll need to symbolicate it yourself.

Symbolication Using the “Symbolicatecrash” Tool

Fortunately, Apple has provided us with a script for retrieving debug symbols and applying them to a crash report.

For Xcode 7.3, you can find the tool at: /Applications/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash

For Xcode 6 through 7.2, you can find the tool at: /Applications/Xcode.app/Contents/SharedFrameworks/DTDeviceKitBase.framework/Versions/Current/Resources/symbolicatecrash

Or if you’re using Xcode 5: /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKitBase.framework/Versions/Current/Resources/symbolicatecrash

To use this tool you’ll need to export the DEVELOPER_DIR environment variable with an appropriate path to your Xcode installation:

export DEVELOPER_DIR=”/Applications/Xcode.app/Contents/Developer”

Place your .crash, .app, and .dSYM files in the same directory and run:

symbolicatecrash ScaryCrash.crash > Symbolicated.crash

You may need to explicitly reference the app binary:

symbolicatecrash ScaryCrash.crash ./Crasher.app/Crasher > Symbolicated.crash

If you’re having trouble, try using the -v verbosity flag before the command line arguments.

Verifying Symbolication

If symbolication didn’t work, double check you grabbed the right .dSYM or .app. You can double check by cross referencing the build UUID in the crash report to the UUID in the app binary:

dwarfdump –uuid Crasher.app/Crasher
UUID: B00CDF0C-2965-3095-B1E8-6078B12D79E5 (armv7) Crasher.app/Crasher UUID: 3F3BE3C6-DD2E-3E23-A603-A18097C9317F (arm64) Crasher.app/Crasher

And in the dSYM:

dwarfdump –uuid Crasher.app.dSYM/Contents/Resources/DWARF/Crasher
UUID: B00CDF0C-2965-3095-B1E8-6078B12D79E5 (armv7) Crasher.app.dSYM/Contents/Resources/DWARF/Crasher UUID: 3F3BE3C6-DD2E-3E23-A603-A18097C9317F (arm64) Crasher.app.dSYM/Contents/Resources/DWARF/Crasher

Compare these UUIDs to your crash report:

0xa8000 – 0xaffff Crasher armv7 <b00cdf0c29653095b1e86078b12d79e5> /var/mobile/Containers/Bundle/Application/956755E3-6C66-4E87-A8BC-352FD4BE3711/Crasher.app/Crasher

The verbose logs from the symbolicatecrash tool also lists the UUIDs it finds.

Troubleshooting the “Symbolicatecrash” Tool

If you’re still mystified, carefully examine the symbolication logs. The symbolication tool attempts to locate appropriate files with matching UUIDs for your app and for each dynamic framework. Look for your app name or UUID and see whether it matched.

…….fetching symbol file for Crasher–[undef] Searching []…– NO MATCH Searching in Spotlight for dsym with UUID of b00cdf0c29653095b1e86078b12d79e5 ... Number of symbols in /Users/You/Workspace/Crasher.app/Crasher: 1 + 106 = 107 Found executable /Users/You/Workspace/Crasher.app/Crasher — MATCH

Here’s an example of a log you might encounter if Spotlight is unable to locate your .dSYM:

Did not find executable for dsym Warning: Can’t find any unstripped binary that matches version of /private/var/mobile/Containers/Bundle/Application/956755E3-6C66-4E87-A8BC-352FD4BE3711/Crasher.app/Crasher

Or if you have an invalid .dSYM:

Number of symbols in ./Crasher: + = 0 ./Crasher appears to be stripped, skipping.

Or any other invalid input:

fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: can’t figure out the architecture type of: ./Crasher.app.dSYM.zip ./Crasher.app.dSYM.zip doesn’t contain armv7 slice

The Xcode 6 version of the symbolicatecrash tool attempts to fix some Spotlight issues that the Xcode 5 version struggled with. If you’re having trouble, it could be a file indexing issue with Spotlight. Try:

mdimport -g /Applications/Xcode.app/Contents/Library/Spotlight/uuid.mdimporter .
Using the Command Line Toolchain

We can take a step deeper and use the developer command line toolchain to symbolicate the memory addresses of our stacktrace line by line. If you’ve had trouble using this method in the past it may be because the .crash formatting has changed in recent years.

First, let’s take another look at the stacktrace in our crash report:

... 13 Crasher 0x000aeef6 0xa8000 + 28406 ...

The leftmost hex value, 0x000aeef6, is the stack address. The second hex value, 0xa8000, is the application load address. The number proceeding the plus sign, 28406, is the decimal value of the subtraction between the stack address and the load address. 0x000aeef6 = 0xa8000 + 0x6EF6. You’ll notice that underneath “Binary Images” in our crash report is a list of dynamic libraries with a range of memory addresses. The starting address of our binary matches the load address in the stacktrace.

Binary Images: 0xa8000 – 0xaffff Crasher armv7 /var/mobile/Containers/Bundle/Application/956755E3-6C66-4E87-A8BC-352FD4BE3711/Crasher.app/Crasher

Next, for good measure, we’ll verify that our executable contains the architecture for our crash using either file or lipo -info:

file Crasher.app/Crasher
Crasher.app/Crasher: Mach-O universal binary with 2 architectures Crasher.app/Crasher (for architecture armv7): Mach-O executable arm Crasher.app/Crasher (for architecture arm64): Mach-O 64-bit executable

Now we are armed with everything we need. We will use atos to convert our addresses to debug symbols. Notice how we provide the load address followed by the stack address and that we specify the architecture which crashed:

atos -arch armv7 -o Crasher.app/Crasher -l 0xa8000 0x000aeef6
main (in Crasher) (main.m:14)

That’s it. If you’re interesting in delving in further, read up on the Mach-O object file format and check out the family of Mach-O command line utilities, namely otool and lipo.

Further Reading

For more comprehensive documentation, see:

Happy symbolicating!

Christopher Hale

Christopher Hale

Christopher is a multidisciplinary software engineer and a bouldering enthusiast. You can follow him on Twitter.
Article


Add your voice to the discussion: