c++ - How to write a custom native visualizer DLL for Visual Studio 2012 debugger?

Question

What is needed to write a custom native visualizer DLL in C++ for Visual Studio 2012 debugger? I want to display a value that can only be calculated from a class/struct on-demand hence a native visualizer DLL is required. Visual Studio 2012 uses a new method for implementing native visualizers called Natvis. As of this time, there is very little correct information on Natvis and especially on using Natvis to call a visualizer DLL. The DLL will calculate a display string based on class/struct member values.

Answer 1

Here's the C++ code that comprises the AddIn DLL. I named the file NatvisAddIn.cpp and the project created NatvisAddIn.dll.

#include "stdafx.h"
#include <iostream>
#include <windows.h>

#define ADDIN_API __declspec(dllexport)

typedef struct tagDEBUGHELPER
{
    DWORD dwVersion;
    HRESULT (WINAPI *ReadDebuggeeMemory)( struct tagDEBUGHELPER *pThis, DWORD dwAddr, DWORD nWant, VOID* pWhere, DWORD *nGot );
    // from here only when dwVersion >= 0x20000
    DWORDLONG (WINAPI *GetRealAddress)( struct tagDEBUGHELPER *pThis );
    HRESULT (WINAPI *ReadDebuggeeMemoryEx)( struct tagDEBUGHELPER *pThis, DWORDLONG qwAddr, DWORD nWant, VOID* pWhere, DWORD *nGot );
    int (WINAPI *GetProcessorType)( struct tagDEBUGHELPER *pThis );
} DEBUGHELPER;

typedef HRESULT (WINAPI *CUSTOMVIEWER)( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved );

extern "C" ADDIN_API HRESULT MyClassFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved );
extern "C" ADDIN_API HRESULT MyStructFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved );

class MyClass
{
public:
    int publicInt;
};

struct MyStruct { int i; };

ADDIN_API HRESULT MyClassFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    MyClass c;
    DWORD nGot;
    pHelper->ReadDebuggeeMemory(pHelper,dwAddress,sizeof(MyClass),&c,&nGot);
    sprintf_s(pResult,max,"Dll MyClass: max=%d nGot=%d MyClass=%x publicInt=%d",max,nGot,dwAddress,c.publicInt);
    return S_OK;
}

ADDIN_API HRESULT MyStructFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    MyStruct s;
    DWORD nGot;
    pHelper->ReadDebuggeeMemory(pHelper,dwAddress,sizeof(MyStruct),&s,&nGot);
    sprintf_s(pResult,max,"Dll MyStruct: max=%d nGot=%d MyStruct=%x i=%d",max,nGot,dwAddress,s.i);
    return S_OK;
}

Here is the .natvis file which Visual Studio 2012 debugger uses to display the value. Place it in a .natvis file. I named it NatvisAddIn.natvis. The file instructs VS 2012 debugger to call NatvisAddIn.dll. The dll contains two visualizer method calls; MyClassFormatter to format MyClass and MyStructFormatter to format MyStruct. The debugger will show the method's formatted value in the Auto, Watch or tooltip display for each instance of the specified type (MyClass, MyStruct).

<?xml version="1.0" encoding="utf-8"?>
    <AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
    <Type Name="MyClass">
        <DisplayString LegacyAddin="NatvisAddIn.dll" Export="MyClassFormatter"></DisplayString>
    </Type>
    <Type Name="MyStruct">
        <DisplayString LegacyAddin="NatvisAddIn.dll" Export="MyStructFormatter"></DisplayString>
    </Type>
</AutoVisualizer>

Place both the compiled NatvisAddIn.dll file and the NatvisAddIn.natvis files into one of the following three locations:

%VSINSTALLDIR%Common7PackagesDebuggerVisualizers (requires admin access)

%USERPROFILE%My DocumentsVisual Studio 2012Visualizers

VS extension folders

You will need to make sure the following registry key exists and the value is 1:

[HKEY_CURRENT_USERSoftwareMicrosoftVisualStudio11.0_ConfigDebugger]

"EnableNatvisDiagnostics"=dword:00000001

If all goes well, you will see natvis messages appear in Visual Studio's debugger Output window. The messages will show whether Natvis was able to parse the .natvis files. Results of parsing every .natvis file is shown in the output window. If something is wrong, use the command "dumpbin/exports " to double check that the DLL methods' names are exactly matching the .navis file's Type=. Also make sure the current .dll and .natvis files have been copied to the appropriate directory.

Natvis: Parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizersatlmfc.natvis.
Natvis: Done parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizersatlmfc.natvis.
Natvis: Parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizersconcurrency.natvis.
Natvis: Done parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizersconcurrency.natvis.
Natvis: Parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizersNatvisAddIn.natvis.
Natvis: Done parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizersNatvisAddIn.natvis.
Natvis: Parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizersstl.natvis.
Natvis: Done parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizersstl.natvis.
Natvis: Parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizerswindows.natvis.
Natvis: Done parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizerswindows.natvis.
Natvis: Parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizerswinrt.natvis.
Natvis: Done parsing natvis xml file: C:Program Files (x86)Microsoft Visual Studio 11.0Common7PackagesDebuggerVisualizerswinrt.natvis.

Test program:

#include "stdafx.h"
#include <iostream>

class MyClass
{
public:
    int publicInt;
};

struct MyStruct { int i; };

int _tmain(int argc, _TCHAR* argv[])
{
    struct MyStruct s = {1234};
    std::cout << s.i << std::endl;
    MyClass *c = new MyClass;
    c->publicInt = 1234;
    std::cout << c->publicInt << std::endl;
    return 0;
}

Information resources:

XmlSchemas atvis.xsd

http://code.msdn.microsoft.com/windowsdesktop/Writing-type-visualizers-2eae77a2

http://blogs.msdn.com/b/mgoldin/archive/2012/06/06/visual-studio-2012-and-debugger-natvis-files-what-can-i-do-with-them.aspx

http://blogs.msdn.com/b/vcblog/archive/2012/07/12/10329460.aspx