Newbie Win32 Development Using Python
If any code snippets or content appear broken or missing, you can find this same post with its full content at blog.foundev.pro.
First a disclaimer. I’m not in anyway shape or form an experienced win32 systems programmer. I’ve always done application development or systems administration scripting. That in-between area where you get out your C/C++ compiler and start dealing with pointers is completely alien to me. This has created problems in the past when I need to do something outside of the scope of C# and found myself staring at MSDN docs in C trying to extrapolate the equivalent C# code, and typically being disgusted at having to use hand rolled Structs with calls to Marshall.GetLastWin32Error().
Worse still I end up with a nasty implementation that requires slow, fragile integration tests to verify behavior or very verbose mirror interfaces where I’m testing order of calls. Look at the following code sample to read a reparse point’s target directory from code project:
// Allocate a buffer for the reparse point data:
Int32 outBufferSize = Marshal.SizeOf(typeof(REPARSE_GUID_DATA_BUFFER));
IntPtr outBuffer = Marshal.AllocHGlobal(outBufferSize);
try
{
// Read the reparse point data:
Int32 bytesReturned;
// WOW what a signature, it’s requiring blank data!
// Looking at MSDN the signaure contains input and output buffer and size
// for both, meaning I’m always sending something as IntPtr.Zero
Int32 readOK = DeviceIoControl( hFile,FSCTL_GET_REPARSE_POINT, IntPtr.Zero, 0,
outBuffer, outBufferSize, out bytesReturned,IntPtr.Zero);
if (readOK != 0)
{
// Get the target directory from the reparse
// point data:
REPARSE_GUID_DATA_BUFFER rgdBuffer =
(REPARSE_GUID_DATA_BUFFER)Marshal.PtrToStructure(outBuffer, typeof(REPARSE_GUID_DATA_BUFFER));
targetDir = Encoding.Unicode.GetString( rgdBuffer.PathBuffer,rgdBuffer.SubstituteNameOffset,
rgdBuffer.SubstituteNameLength);
Int32 outBufferSize = Marshal.SizeOf(typeof(REPARSE_GUID_DATA_BUFFER));
IntPtr outBuffer = Marshal.AllocHGlobal(outBufferSize);
try
{
// Read the reparse point data:
Int32 bytesReturned;
// WOW what a signature, it’s requiring blank data!
// Looking at MSDN the signaure contains input and output buffer and size
// for both, meaning I’m always sending something as IntPtr.Zero
Int32 readOK = DeviceIoControl( hFile,FSCTL_GET_REPARSE_POINT, IntPtr.Zero, 0,
outBuffer, outBufferSize, out bytesReturned,IntPtr.Zero);
if (readOK != 0)
{
// Get the target directory from the reparse
// point data:
REPARSE_GUID_DATA_BUFFER rgdBuffer =
(REPARSE_GUID_DATA_BUFFER)Marshal.PtrToStructure(outBuffer, typeof(REPARSE_GUID_DATA_BUFFER));
targetDir = Encoding.Unicode.GetString( rgdBuffer.PathBuffer,rgdBuffer.SubstituteNameOffset,
rgdBuffer.SubstituteNameLength);
//removed a bunch more stuff to format the result
// Free the buffer for the reparse point data:
Marshal.FreeHGlobal(outBuffer);
<p>
 
</p>
<p>
In the past when building these kind of things for less complicated code I just would mirror out all the untestable code like Marshal , DeviceIoControl and CloseHandle calls with interfaces, then create a production wrapper.  This however leads to very long tests, with lots of specifics about the internals of the interaction, in the above example the support code looks like:
</p>
<div style="padding-bottom: 0px;margin: 0px;padding-left: 0px;padding-right: 0px;float: none;padding-top: 0px" class="wlWriterEditableSmartContent">
<div style="font-family:consolas,lucida console,courier,monospace">
    <span style="color:#0000ff">public</span> <span style="color:#0000ff">interface</span> IMarshal<br />     <span style="color:#0000ff">{</span></p>
<p>
        <span style="color:#0000ff">void</span> FreeHGlobal(IntPtr buffer);<br />         Object PtrToStructure(IntPtr pointer, Type structure);<br />         Int32 SizeOf(Type structure);<br />         IntPtr AllocHGlobal(<span style="color:#2b91af">int</span> cb);<br />     <span style="color:#0000ff">}</span>
</p>
<p>
    <span style="color:#0000ff">public</span> <span style="color:#0000ff">class</span> <span style="color:#2b91af">ProdMarshal</span> : IMarshal<br />     <span style="color:#0000ff">{</span><br />         <span style="color:#0000ff">public</span> <span style="color:#0000ff">void</span> FreeHGlobal(IntPtr buffer)<span style="color:#0000ff">{</span><br />         <br />             Marshal.FreeHGlobal(buffer);<br />         <span style="color:#0000ff">}</span>
</p>
<p>
        <span style="color:#008000">// you get the idea from here <br /> </span>    <span style="color:#0000ff">}</span> </div> </div>
<p>
Not terribly pretty mocking and testing this leads to is it?  This isn’t also counting the win32 calls that are needed to setup and finalize the data around this. Before its all said and done to make this “testable” you end up with 2 interfaces, 2 wrapper production classes, and very heavy interaction specific mocking to even handle this basic idea of getting a value from a file system object.  Before you try and point out “well you can reuse that logic later” look at the size of Marshal and ask yourself do you really want to mock all that out?
</p>
<p>
Adding insult to injury your mocks have to be very good to replicate the behavior of the actual real thing, and it varies greatly across platforms. Windows 7 and server 2008 r2 allow reparse points (or symlinks in the this case) from directory to directory, but Vista and 2008 r1 are a no go and will fail.
</p>
<p>
No wonder the systems developers that I know completely turn there nose up at unit testing and instead focus on slow integration or systems tests. The abstractions take time to make happen and the code itself is just not testable without lots of mirrored interfaces.
</p>
<p>
So not wanting to spend all my time just writing mirrored interfaces day in and out for win32 calls, nor liking the idea of turning my back on unit testing, I decided to take a look at dynamic languages win32 support.
</p>
<p>
First I looked at Ruby and the win32 libraries I saw didn’t give me the above functionality I needed with the minimal amount of hassle I wanted. DeviceIoControl up there for example with its stunningly bad signature is still pretty rough in Ruby as they have a one to one signature mapping. 
</p>
<p>
Python however has had for a very long time strong win32api support and removes some of the awful pain that you have to deal with. Below is the equivalent of the code above, again with items stripped out for brevity before and after it, but they’re roughly computationally equivalent and the Python version wins out more the more I include.
</p>
<p>
 
</p>
<div style="padding-bottom: 0px;margin: 0px;padding-left: 0px;padding-right: 0px;float: none;padding-top: 0px" class="wlWriterEditableSmartContent">
<div style="font-family:consolas,lucida console,courier,monospace">
<span style="color:#000000">output_buf</span><span style="color:#ce5c00"><b>=</b></span><span style="color:#000000">win32file</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">AllocateReadBuffer</span><span style="color:#000000"><b>(</b></span><span style="color:#000000">winnt</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">MAXIMUM_REPARSE_DATA_BUFFER_SIZE</span><span style="color:#000000"><b>)</b></span> <br /> <span style="color:#8f5902"><i>#notice how I don’t have to pass in size of the buffers now. I still have the need to </i></span><br /> <span style="color:#8f5902"><i># pass None in for the input buffer here however</i></span><br /> <span style="color:#000000">buf</span><span style="color:#ce5c00"><b>=</b></span><span style="color:#000000">win32file</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">DeviceIoControl</span><span style="color:#000000"><b>(</b></span><span style="color:#000000">h</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">winioctlcon</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">FSCTL_GET_REPARSE_POINT</span><span style="color:#000000"><b>,</b></span><span style="color:#3465a4">None</span><span style="color:#000000"><b>,</b></span><span style="color:#000000">OutBuffer</span><span style="color:#ce5c00"><b>=</b></span><span style="color:#000000">output_buf</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">Overlapped</span><span style="color:#ce5c00"><b>=</b></span><span style="color:#3465a4">None</span><span style="color:#000000"><b>)</b></span> <br />         <br /> <span style="color:#000000">fixed_fmt</span><span style="color:#ce5c00"><b>=</b></span><span style="color:#4e9a06">‘LHHHHHH’</span> <br /> <span style="color:#000000">fixed_len</span><span style="color:#ce5c00"><b>=</b></span><span style="color:#000000">struct</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">calcsize</span><span style="color:#000000"><b>(</b></span><span style="color:#000000">fixed_fmt</span><span style="color:#000000"><b>)</b></span> <br /> <span style="color:#000000">tag</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">datalen</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">reserved</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">target_offset</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">target_len</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">printname_offset</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">printname_len</span> <span style="color:#ce5c00"><b>=</b></span>  <br />      <span style="color:#000000">struct</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">unpack</span><span style="color:#000000"><b>(</b></span><span style="color:#000000">fixed_fmt</span><span style="color:#000000"><b>,</b></span> <span style="color:#000000">buf</span><span style="color:#000000"><b>[:</b></span><span style="color:#000000">fixed_len</span><span style="color:#000000"><b>])</b></span> 
</div>
</div>
<p>
 
</p>
<p>
It’s still not pretty, but its briefer, and I can actually stub out or test struct.calcsize, AllocateReadBuffer and DeviceIoControl with no need to mess around with setting up lots of test harness code. Using my current mocking framework the test code looks something like this:
</p>
<p>
 
</p>
<div style="padding-bottom: 0px;margin: 0px;padding-left: 0px;padding-right: 0px;float: none;padding-top: 0px" class="wlWriterEditableSmartContent">
<div style="font-family:consolas,lucida console,courier,monospace">
<span style="color:#000000">alloc</span> <span style="color:#ce5c00"><b>=</b></span> <span style="color:#000000">mock</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">replaceWithMock</span><span style="color:#000000"><b>(</b></span><span style="color:#000000">win32file</span><span style="color:#000000"><b>,</b></span> <span style="color:#4e9a06">“AllocateReadBuffer”</span><span style="color:#000000"><b>)</b></span><br /> <span style="color:#204a87">buffer</span> <span style="color:#ce5c00"><b>=</b></span> <span style="color:#204a87">object</span><span style="color:#000000"><b>()</b></span><br /> <span style="color:#000000">alloc</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">stub</span><span style="color:#000000"><b>(</b></span><span style="color:#204a87">buffer</span><span style="color:#000000"><b>)</b></span></p>
<p>
<span style="color:#8f5902"><i>#removed extra setup code for calcsize and DeviceIoControl</i></span>
</p>
<p>
<span style="color:#000000">getTargetDir</span><span style="color:#000000"><b>()</b></span>
</p>
<p>
<span style="color:#000000">mock</span><span style="color:#ce5c00"><b>.</b></span><span style="color:#000000">assertCalled</span><span style="color:#000000"><b>(</b></span><span style="color:#000000">win32file</span><span style="color:#000000"><b>,</b></span> <span style="color:#4e9a06">“AllocateReadBuffer”</span><span style="color:#000000"><b>)</b></span> </div> </div>
<p>
 
</p>
<p>
Much cleaner, much lower friction and so I’m satisfied with Python for win32 development so far.  I’ll give C# another look when I have the dynamic keyword and I don’t have to make explicit mirror interfaces for everything that’s untestable.
</p>