使用 C++ 和多线程在交付的构建中定位错误

Question

我们将我们的软件版本交付给客户，他们在由于数据隐私而无法共享的数据上运行代码。因此，我们无法正常调试导致错误（表现为超时、异常和崩溃）的数据。相反，当发生错误时，我会得到一个异常地址，然后我需要将其映射到代码位置。我们目前使用 MSVC 编译 C++ 代码部分，但 MSVC 不再允许创建映射文件，我可以在其中找到产生错误的源代码行 (How can I create a map file with line numbers in Visual C++ 2005?)。是否有任何 C++ 编译器（适用于 Windows、x86 和 x64）可以配置为生成具有准确行号的映射文件？

另外，我正在寻找 Boost.stacktrace 的解决方案，但我们遇到的另一种情况是，由于另一个线程超时，一个线程必须停止。我不确定我是否可以在 Boost 中为其他线程获取堆栈跟踪...

我尝试的第三件事是保留使用__FILE__ 和__LINE__ 定义设置的两个变量lastFile、lastLineNo。代码中散布着大量的调用

#define SET_OP lastFile = __FILE__; lastLineNo = __LINE__;

这种方法的问题是在 DLL 中引入了 lastFile、lastLineNo，但它们当然应该是线程本地的（延迟加载的 DLL 不能很好地与线程本地存储配合使用）。 Boost.Thread 具有线程本地存储，但我找不到在超时的情况下从另一个线程获取这些变量的方法。许多对 SET_OP 的调用都在需要高性能的函数中，所以我对需要互斥锁的复杂事物有点害羞。

我感觉我正在突破多线程场景中堆栈跟踪的可能性，但这是一个非常现实的问题，现在已经让我头痛了数周，甚至数月。

Answer 1

我更喜欢一种可以生成完整堆栈跟踪的方法，而不是将单个地址转换为源代码位置的方法。我无法解决 boost.stacktrace 不会为其他线程中的无限循环生成堆栈跟踪的问题。尽管使用 boost.stacktrace 可以通过堆栈跟踪抛出异常，但堆栈跟踪不可用的情况太多了。希望 std::backtrace 将使堆栈跟踪可用于所有异常。在那之前，我选择遵循检测源代码方法。它也不完美，因为我们的代码库太大而无法完全检测，而且第三方库也不支持这种检测。所以我想出了这个：

#ifdef _WIN32
#include "Windows.h"
#undef min
#undef max
#endif

#include <algorithm>
#include <iostream>
#include <exception>
#include <map>
#include <mutex>
#include <string>
#include <thread>
#include <vector>

#define StackTraceSize 20
class StackTrace final {
public:
    const char* mLastFileName = 0;
    int mLastLineNo = 0;
    std::string uncleanExits;

    // Constructor
    StackTrace() = default;

    // Destructor
    ~StackTrace() = default;

    // Store information about a scope that was entered
    void enterScope(const char* fileName, int lineNo);

    void update(int lineNo);

    // Forget information about a scope
    void leaveScope();

    // Generate string representation of the whole stack trace
    std::string toString() const;

    // Generate string representation of current stack frame
    std::string getCurrentStackFrameString() const;

private:
    int mCurrNo = -1;
    const char* mFileNames[StackTraceSize];
    int mLineNos[StackTraceSize];
};

////////////////////////////////////////////////////////////////////////////////

std::mutex threadsWithStackTraceMutex;
std::map<std::thread::id, StackTrace*> threadsWithStackTraces;

StackTrace* getStackTrace(std::thread::id threadId)
{
    std::lock_guard<std::mutex> lock(threadsWithStackTraceMutex);
    if (threadsWithStackTraces.count(threadId) == 0) {
        // threadsWithStackTraces[threadId] could fail theoretically, due to out of memory.
        // In that case, the new StackTrace would be leaked, but it probably wouldn't happen often,
        // due to the memory shortage ...
        threadsWithStackTraces[threadId] = new StackTrace();
    }
    // if addThread fails, there would be an exception. Otherwise, the assignment below should be safe.
    StackTrace* result = threadsWithStackTraces[threadId];
    return result;
}

void forgetStackTrace(std::thread::id threadId)
{
    std::lock_guard<std::mutex> lock(threadsWithStackTraceMutex);
    if (threadsWithStackTraces.count(threadId) > 0) {
        delete threadsWithStackTraces[threadId];
        threadsWithStackTraces.erase(threadId);
    }
}

////////////////////////////////////////////////////////////////////////////////
// A simple integer storing the currently active operation - used for debugging
// crashes

// Store information about a scope that was entered
void StackTrace::enterScope(const char* fileName, int lineNo)
{
    if (mCurrNo < 0) {
        mCurrNo = 0;
    } else {
        ++mCurrNo;
    }

    if (mCurrNo < StackTraceSize) {
        mFileNames[mCurrNo] = fileName;
        mLineNos[mCurrNo] = lineNo;
    } else {
        mLastFileName = fileName;
        mLastLineNo = lineNo;
    }
}

// Update line number. Filename doesn't change here ... same scope (normally)
void StackTrace::update(int lineNo)
{
    if (mCurrNo >= 0 && mCurrNo < StackTraceSize) {
        mLineNos[mCurrNo] = lineNo;
    } else {
        mLastLineNo = lineNo;
    }
}

// Forget information about a scope
void StackTrace::leaveScope()
{
    --mCurrNo;

    if (mCurrNo == 0) {
        forgetStackTrace(std::this_thread::get_id());
    }
}

StackTrace* getStackTrace()
{
    StackTrace* result = getStackTrace(std::this_thread::get_id());
    return result;
}

// Generate string representation of current stack frame
std::string StackTrace::getCurrentStackFrameString() const
{
    //SET_SCOPE; // doesn't belong here!!! We don't want this function to appear in stack traces ...
    if (mCurrNo < StackTraceSize) {
        return std::string(mFileNames[mCurrNo]) + std::string(":") + std::to_string(mLineNos[mCurrNo]);
    } else {
        return std::string(mLastFileName) + std::string(":") + std::to_string(mLastLineNo);
    }
}

// Generate string representation of the stack trace
std::string StackTrace::toString() const
{
    //SET_SCOPE; // doesn't belong here!!! We don't want this function to appear in stack traces ...
    std::string result;
    int printNow = std::min(StackTraceSize, mCurrNo+1);
    for (int i = 0; i < printNow; ++i) {
        if (i > 0) {
            result += ", ";
        }
        result += mFileNames[i];
        result += ":";
        result += std::to_string(mLineNos[i]);
    }

    if (mCurrNo > StackTraceSize) {
        result += std::string(", (") + std::to_string(mCurrNo - StackTraceSize) +
            std::string(" unrecorded entries)");
    }
    if (mCurrNo >= StackTraceSize) {
        result += ", ";
        result += mLastFileName;
        result += ":";
        result += std::to_string(mLastLineNo);
    }

    return result;
}

std::string getStackTraceString(std::thread::id thread_id = std::this_thread::get_id())
{
    return getStackTrace(thread_id)->toString();
}

// Simple construct to print a message about missing RETURN statements
class StackInfo final {
public:
    bool cleanExit = false;
    StackTrace* mSt;

    StackInfo(StackTrace* st)
    : mSt(st)
    {
    }

    ~StackInfo()
    {
        if (!cleanExit) {
            if (mSt != nullptr) {
                if (!mSt->uncleanExits.empty()) {
                    mSt->uncleanExits += ",";
                }
                mSt->uncleanExits += mSt->getCurrentStackFrameString();
            }
        }
    }
};

#define SET_SCOPE StackTrace* st = getStackTrace(); st->enterScope(__FILE__, __LINE__); StackInfo stackInfo(st);
#define SET_OP st->update(__LINE__);
#define RETURN stackInfo.cleanExit = true; st->leaveScope(); return 

// For functions that are called very often, it is faster to pass the stack infos as a separate argument ...
#define SET_SCOPE_FAST StackInfo stackInfo(st); st->enterScope(__FILE__, __LINE__);

void logStackTrace()
{
    auto* st = getStackTrace();
    if (st != nullptr) {
        std::string msg = st->toString();
        std::cout << msg << std::endl;
    }
}

///////////////////////////////////////////////////////////////////////////////////////////////
// Demo code

void func1(StackTrace* st, int recursions)
{
    SET_SCOPE_FAST;
    for (int i = 0; i < 500; ++i) {
        SET_OP;
        if (recursions > 0) {
            func1(st, recursions-1);
        }
    }
    RETURN;
}

void func2()
{
    SET_SCOPE;
    func1(st, 2);
    RETURN;
}

class ExceptionWithText: public std::exception {
public:
    std::string mMsg;

    ExceptionWithText(const std::string& msg)
        : mMsg(msg)
    {
    }

    const char* what() const noexcept override
    {
        return mMsg.c_str();
    }
};

void throw_Cpp_exception_with_backtrace()
{
    SET_SCOPE;
    std::string str = "Exception thrown - StackTrace: " + getStackTraceString();
    throw ExceptionWithText(str);
    RETURN;
}

void cause_Cpp_exception_with_no_backtrace()
{
    SET_SCOPE;
    std::vector<int> ints;
    std::cout << ints.at(2) + ints.at(3);
    RETURN;
}

void endless_loop()
{
    SET_SCOPE;
    while(1);
    RETURN;
}

void crash_divByZero()
{
    int i = 1;
    std::cout << i / (i-1);
}

int crash_deref_null()
{
    char* p = nullptr;
    return p[0];
}

int crash_infinite_recursion()
{
    return crash_infinite_recursion();
}

int crash_double_free()
{
    SET_SCOPE;
    char* p = new char[10];
    char* q = p;
    delete p;
    delete q;
    RETURN 0;
}

typedef void (*VoidFunc)();

class Functor {
public:
    bool mRunning = false;
    std::chrono::system_clock::time_point startTime;
    std::thread* thread = nullptr;
    VoidFunc func = nullptr;
    std::string exceptionCaughtStr;

    Functor()
    : mRunning(false)
    {
    }

    void operator()()
    {
        SET_SCOPE;
        try {
            mRunning = true;
            SET_OP;
            func();
        } catch(std::exception e) {
            exceptionCaughtStr = e.what();
            StackTrace* st = getStackTrace();
            if (st != nullptr && !st->uncleanExits.empty()) {
                exceptionCaughtStr +=" - unclean exits: " + st->uncleanExits;
            }
        }
        mRunning = false;
        RETURN;
    }
};

std::string timeDiffStr(const std::chrono::system_clock::time_point& t1)
{
    std::chrono::system_clock::time_point t2 = std::chrono::system_clock::now();
    auto duration = t2-t1;
    auto milliseconds = std::chrono::duration_cast<std::chrono::milliseconds>(duration);
    return std::to_string(milliseconds.count() / 1000.0) + std::string(" s");
}

#ifdef _WIN32
// Function to convert Windows system exceptions into C++ exceptions
void se_trans_func(unsigned int code, EXCEPTION_POINTERS* pExp)
{
    std::string msg = std::string("SEH exception code ")
        + std::to_string(code);

    switch (code) {
        case 0xc0000005: 
            msg += " (access violation)";
            break;
        case 0xc0000094: 
            msg += " (division by zero)";
            break;
    }

    msg += std::string(". Stacktrace: ") + getStackTraceString();
    throw std::exception(msg.c_str());
}
#endif

int main(int argc, char** args)
{
    SET_SCOPE;
    // Catch an exception that includes a backtrace
    try {
        throw_Cpp_exception_with_backtrace();
    } catch(const std::exception& e) {
        std::cout << "Exception caught: " << e.what() << std::endl;
        st->uncleanExits.clear();
    }

    // Store an exception from another thread in a functor's exception text buffer
    try {
        Functor functor;
        functor.mRunning = true;
        functor.func = &throw_Cpp_exception_with_backtrace;
        functor.thread = new std::thread(std::ref(functor));
        while (functor.mRunning);
        functor.thread->join();
        delete functor.thread;
        std::cout << "Functor exception: " << functor.exceptionCaughtStr << std::endl;
    } catch(const std::exception& e) {
        std::cout << "Exception caught: " << e.what() << std::endl;
    }

    // Catch an exception that includes a backtrace
    try {
        cause_Cpp_exception_with_no_backtrace();
    } catch(const std::exception& e) {
        std::cout << "Exception caught: " << e.what() << std::endl;
        StackTrace* st = getStackTrace();
        if (st != nullptr && !st->uncleanExits.empty()) {
            std::cout << "Unclean exits: " + st->uncleanExits << std::endl;
            st->uncleanExits.clear();
        }
    } catch(...) {
        std::cout << "An unknown Exception was caught. " << std::endl;
    }

    // Store an exception from another thread in a functor's exception text buffer
    try {
        Functor functor;
        functor.mRunning = true;
        functor.func = &cause_Cpp_exception_with_no_backtrace;
        functor.thread = new std::thread(std::ref(functor));
        while (functor.mRunning);
        functor.thread->join();
        delete functor.thread;
        std::cout << "Functor exception: " << functor.exceptionCaughtStr << std::endl;
    } catch(const std::exception& e) {
        std::cout << "Exception caught: " << e.what() << std::endl;
    }

    // See if we can get a stacktrace for another thread that hit an endless loop
    try {
        Functor functor;
        functor.mRunning = true;
        functor.func = &endless_loop;
        functor.thread = new std::thread(std::ref(functor));
        std::this_thread::sleep_for(std::chrono::seconds(1));
        if (functor.mRunning) {
            std::cout << "Stack trace for other thread in endless loop: " 
                      << getStackTraceString(functor.thread->get_id())
                      << std::endl;
        } else {
            std::cout << "Why did the endless loop end?" << std::endl;
        }
    } catch(std::exception e) {
        std::cout << "Exception caught: " << e.what() << std::endl;
    }

    // Test Windows exceptions
    // todo: this exception is treated correctly, but a previous entry remained in the stack trace
    // (cause_Cpp_exception_with_no_backtrace)
    #ifdef _WIN32
    SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOALIGNMENTFAULTEXCEPT | SEM_NOGPFAULTERRORBOX);
    _set_se_translator(se_trans_func);

    try {
        crash_divByZero();
    } catch(const std::exception& e) {
        std::cout << e.what() << std::endl;
    }
    #endif

    try {
        // Test performance with a single thread
        {
            auto t1 = std::chrono::system_clock::now();
            func2();
            std::cout << "Program phase 1 duration: " << timeDiffStr(t1) << std::endl;
        }

        // Now, test performance with several threads
        {
            size_t threads = 4;
            std::vector<Functor> functorVector;
            for (size_t i = 0; i < threads; ++i) {
                functorVector.emplace_back(Functor());
            }
            int endedThreads = 0;
            while (endedThreads < 8) {
                for (size_t i = 0; i < threads; ++i) {
                    Functor& functor = functorVector[i];
                    if (functor.thread == nullptr) {
                        functor.mRunning = true;
                        functor.func = &func2;
                        functor.thread = new std::thread(std::ref(functor));
                        functor.startTime = std::chrono::system_clock::now();
                    } else if (!functor.mRunning) {
                        functor.thread->join();
                        delete functor.thread;
                        functor.thread = nullptr;
                        std::cout << "Functor " << i << " duration: " << timeDiffStr(functor.startTime) << std::endl;
                        ++endedThreads;
                    }
                }
            }
        }
    } catch(std::exception e) {
        std::cout << "Exception " << e.what() << std::endl;
    }
    RETURN 1;
}

这会产生输出：

Exception caught: Exception thrown - StackTrace: C:\Temp\stackTraceTest\main.cpp:362, C:\Temp\stackTraceTest\main.cpp:249
Functor exception: Unknown exception - unclean exits: C:\Temp\stackTraceTest\main.cpp:249
Exception caught: invalid vector subscript
Unclean exits: C:\Temp\stackTraceTest\main.cpp:257
Functor exception: invalid vector subscript - unclean exits: C:\Temp\stackTraceTest\main.cpp:257
Stack trace for other thread in endless loop: C:\Temp\stackTraceTest\main.cpp:317, C:\Temp\stackTraceTest\main.cpp:265
SEH exception code 3221225620 (division by zero). Stacktrace: C:\Temp\stackTraceTest\main.cpp:362, C:\Temp\stackTraceTest\main.cpp:249, C:\Temp\stackTraceTest\main.cpp:257
Program phase 1 duration: 3.867000 s
Functor 1 duration: 5.182000 s
Functor 0 duration: 5.236000 s
Functor 2 duration: 5.291000 s
Functor 3 duration: 5.319000 s
Functor 1 duration: 5.128000 s
Functor 0 duration: 5.179000 s
Functor 2 duration: 5.138000 s
Functor 3 duration: 5.240000 s

仍然非常欢迎评论。