Paul Shinn

Your sample works if you populate a DataSet instead of a DataTable.

Here is a copy of your source with the minimum changes required. Note that when you're using a DataSet you should add code to check whether any tables were returned, and whether there are rows in the first table available, etc.

Caller:

SqlParameter[] Parameters = new SqlParameter[1];
Parameters[0] = new SqlParameter();
Parameters[0].ParameterName = "@TestId";
Parameters[0].Value = TestId;
Parameters[0].SqlDbType = SqlDbType.Int;
Parameters[0].Size = 50;

DataSet data = ExecuteDataSet("ExportTestAsXML", Parameters);

// Read First table (Tables[0]), First Row (Rows[0]), First Column of that Row (Rows[0][0])
System.Diagnostics.Debug.Write(data.Tables[0].Rows[0][0]);

Method:

private DataSet ExecuteDataSet(string storedProcName, SqlParameter[] parameters)
{
    SqlCommand command = new SqlCommand();
    command.CommandText = storedProcName;
    command.Parameters.AddRange(parameters);
    command.CommandType = CommandType.StoredProcedure;
    command.Connection = (SqlConnection)dcMUPView.Connection;

    command.Connection.Open();
    command.Prepare();

    SqlDataAdapter adapter = new SqlDataAdapter(command);
    DataSet ds = new DataSet ();
    adapter.Fill(ds);

    command.Connection.Close();
    return ds;
}

The default packing is along 8-byte boundaries, according to the MIDL command line switch reference here:

/Zp switch @ MSDN (MIDL Language Reference)

Other parts of your code are more likely to break first if the pack value is changed, as the IDL file is usually pre-compiled ahead of time, and it is rare that someone will deliberately alter the command line switches given to MIDL (but not so rare that someone could fiddle with the C-scope #pragma pack and forget to restore the default state).

If you have a good reason to alter the setting, you can explicitly set the packing with a pragma pack statement.

pragma Attribute @ MSDN (MIDL Language Reference)

It is pretty good fortune that no party has changed any setting that would interfere with the default packing. Can it go wrong? Yes, if someone goes out of their way to change the defaults.

When using an IDL file, the details are typically compiled into a typelib (.tlb), and it is assumed that the platform are the same for both servers and clients when using the same typelib. This is suggested in the footnotes for the /Zp switch, as certain values will fail against certain non-x86 or 16-bit targets. There can also be 32bit <-> 64bit conversion cases that could cause expectations to break. Unfortunately I don't know if there are even more cases out there, but the defaults do work with minimal fuss.

C# and VB do not have any intrinsic behavior to handle information in a .tlb; instead, a tool like tlbimp is typically used to convert COM definitions into definitions usable from .NET. I can't verify whether all expectations succeed between C#/VB.NET and COM clients and servers; However, I can verify that using a specific pragma setting other than 8 will work if you reference a .tlb that was created from an IDL compiled under that setting. While I wouldn't recommend going against the default pragma pack, here are the steps to perform if you'd like a working example to use as a reference. I created a C++ ATL project and a C# project to check.

Here are the C++ side instructions.

1. I created an ATL project called SampleATLProject with the default settings in Visual Studio 2010, no fields changed. This should create a dll project for you.
2. Compiled the project to assure that the proper C-side interface files are being created (SampleATLProject_i.c and SampleATLProject_i.h).
3. I added an ATL Simple Object called SomeFoo to the project. Again, no defaults were altered. This creates a class called CSomeFoo that is added to your project.
4. Compile SampleATLProject.
5. I right-clicked the SampleATLProject.idl file, then under the MIDL settings, set the Struct Member Alignment to 4 bytes (/Zp4).
6. Compile SampleATLProject.
7. I altered the IDL to add a struct definition called 'BarStruct'. This entailed adding a C-style struct definition with the MIDL uuid attribute, and an entry in library section referencing the struct definition. See snippet below.
8. Compile SampleATLProject.
9. From the Class View, I right-clicked on ISomeFoo and added a method called FooIt, that takes a struct BarStruct as an [in] parameter called theBar.
10. Compile SampleATLProject.
11. In SomeFoo.cpp, I added some code to print out the size of the struct and throw up a Message Box containing the details.

Here is my IDL for the ATL project.

import "oaidl.idl";
import "ocidl.idl";

[uuid(D2240D8B-EB97-4ACD-AC96-21F2EAFFE100)]
struct BarStruct
{
  byte a;
  int b;
  byte c;
  byte d;
};

[
  object,
  uuid(E6C3E82D-4376-41CD-A0DF-CB9371C0C467),
  dual,
  nonextensible,
  pointer_default(unique)
]
interface ISomeFoo : IDispatch{
  [id(1)] HRESULT FooIt([in] struct BarStruct theBar);
};
[
  uuid(F15B6312-7C46-4DDC-8D04-9DEA358BD94B),
  version(1.0),
]
library SampleATLProjectLib
{
  struct BarStruct;
  importlib("stdole2.tlb");
  [
    uuid(930BC9D6-28DF-4851-9703-AFCD1F23CCEF)      
  ]
  coclass SomeFoo
  {
    [default] interface ISomeFoo;
  };
};

Inside the CSomeFoo class, here is the implementation for FooIt().

STDMETHODIMP CSomeFoo::FooIt(struct BarStruct theBar)
{
  WCHAR buf[1024];
  swprintf(buf, L"Size: %d, Values: %d %d %d %d", sizeof(struct BarStruct), 
           theBar.a, theBar.b, theBar.c, theBar.d);

  ::MessageBoxW(0, buf, L"FooIt", MB_OK);

  return S_OK;
}

Next, on the C# side:

1. Go to the debug or desired output directory for SampleATLProject and run tlbimp.exe on the .tlb file generated as part of the C++ project output. The following worked for me:

   tlbimp SampleATLProject.tlb /out:Foo.dll /namespace:SampleATL.FooStuff

2. Next, I created a C# console application, and added a reference to Foo.dll to the project.

3. In the References folder, go to the Properties for Foo and turn off Embed Interop Types by setting it to false.
4. I added a using statement to reference the namespace SampleATL.FooStuff as given to tlbimp, added the [STAThread] attribute to Main() (the COM apartment models have to match for in-proc consumption), and added some code to call the COM component.

Tlbimp.exe (Type Library Importer) @ MSDN

Here is the source code for that console app.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using SampleATL.FooStuff;

namespace SampleATLProjectConsumer
{
    class Program
    {
        [STAThread]
        static void Main(string[] args)
        {
            BarStruct s;
            s.a = 1;
            s.b = 127;
            s.c = 255;
            s.d = 128;

            ISomeFoo handler = new SomeFooClass();
            handler.FooIt(s);
        }
    }
}

Finally, it runs and I get a modal popup with the following string displayed:

Size: 12, Values: 1 127 255 128

To be sure that a pragma pack change can be made (as 4/8 byte packing are the most common alignments used), I followed these steps to change it to 1:

1. I returned to the C++ project, went to the properties for SampleATLProject.idl and changed the Struct Member Alignment to 1 (/Zp1).
2. Recompile SampleATLProject
3. Run tlbimp again with the updated .tlb file.
4. A warning icon will appear on the .NET File Reference to Foo, but may disappear if you click on the reference. If it doesn't, you can remove and re-add the reference to the C# console project to be sure it is using the new updated version.

I ran it from here and got this output:

Size: 12, Values: 1 1551957760 129 3

That's weird. But, if we forcefully edit the C-level pragma in SampleATLProject_i.h, we get the correct output.

#pragma pack(push, 1)
/* [uuid] */ struct  DECLSPEC_UUID("D2240D8B-EB97-4ACD-AC96-21F2EAFFE100") BarStruct
    {
    byte a;
    int b;
    byte c;
    byte d;
    } ;
#pragma pack(pop)

SampleATLProject is recompiled here, no changes to the .tlb or .NET project, and we get the following:

Size: 7, Values: 1 127 255 128

Regarding IDispatch, it depends on whether your client is late-bound. Late-bound clients have to parse the type information side of IDispatch and discern the proper definitions for non-trivial types. The documentation for ITypeInfo and TYPEATTR suggests that it is possible, given that the cbAlignment field provides the information necessary. I suspect most will never alter or go against the defaults, as this would be tedious to debug if things went wrong or if the pack expectations had to change between versions. Also, structures are not typically supported by many scripting clients that can consume IDispatch. One can frequently expect that only the types governed by the IDL oleautomation keyword are supported.

IDispatch interface @ MSDN
IDispatch::GetTypeInfo @ MSDN
ITypeInfo interface @ MSDN
TYPEATTR structure @ MSDN

oleautomation keyword @ MSDN

I don't think there is a trivial way to resolve the actual owner of a Mutex, but the process that owns it can create other secondary items whose lifetimes are tied to it. There are plenty of mechanisms that are suitable for calling back across-process without having a main window.

1. Register an object in the COM Running Object Table. Clients that are unable to take ownership of the Mutex can lookup the owner via the ROT and call back to the owner. A File Moniker should be suitable for registration here.
2. Create a chunk of shared memory containing location details for the owner process. From there, write into the buffer the process handle and thread handle of a thread that can receive windows messages, and then use PostThreadMessage() to send a notification. Any other competing process may open the shared memory for read-only to determine where to send a windows message.
3. Listen in the owner process on a Socket or Named Pipe. Probably overkill and not a good match for your needs.
4. Use a shared file with locking. I'm not fond of this because the owner will need to poll, and it won't gracefully handle N potential other processes that could be trying to contact the owner at the same time.

Here are reference links for the first two options.

1. IRunningObjectTable @ MSDN , File Monikers @ MSDN
2. Creating Named Shared Memory @ MSDN

MessageBox() has a windows message pump cycle built into it to service window messages.

Somewhere in your application at a very low level within your design, you need a windows message loop to service messages for the window hosting your openGL content. This should run from within the thread that created the window. The ill effects you are seeing in other places could easily be caused by contention side-effects when MessageBox() is invoked from a different thread.

Here is an intro to the operation of window messages at MSDN.

Using Messages and Message Queues @ MSDN

Very simply put (via a code sample from wikipedia), you need a standing loop akin to the following:

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow)
{
    MSG msg;
    while(GetMessage(&msg, NULL, 0, 0) > 0)
    {
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }
    return msg.wParam;
}

Depending on how you have handled updating the window, if you are using paint messages (WM_PAINT), you need to ensure these are dispatched as quickly as possible, or you may need to suppress them entirely if you are drawing direct to the the device DC. I'm mentioning this because you may find additional bugs once a message loop is properly setup.

WM_PAINT message @ MSDN

The sample you've provided doesn't match the members of the class TwitterProfile. Notwithstanding this, you don't need a helper class to handle an array. You can simply target the array type directly at deserialization.

Here's a code sample that you can use to test. I've removed your TwitterProfiles class and stripped down TwitterProfile to just a field named CreatedAt.

Note the following line in the source below:

DataContractJsonSerializer js = 
    new DataContractJsonSerializer(typeof(TwitterProfile[]));

Here is the rest.

using System.Windows.Forms;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Json;
using System.IO;

namespace TwProfileJson
{
    class Program
    {
        [STAThread]
        static void Main(string[] args)
        {
            OpenFileDialog dlg = new OpenFileDialog();
            if (dlg.ShowDialog() != DialogResult.OK) { return; }
            string json = System.IO.File.ReadAllText(dlg.FileName);

            using(MemoryStream stm = new MemoryStream(Encoding.UTF8.GetBytes(json)))
            {
                DataContractJsonSerializer js = new DataContractJsonSerializer(typeof(TwitterProfile[]));
                TwitterProfile[] pps = (TwitterProfile[])js.ReadObject(stm);

                foreach(TwitterProfile p in pps)
                {
                    Console.WriteLine(p.CreatedAt);
                }
            }

            Console.ReadKey();
        }

        [DataContract]
        public class TwitterProfile
        {
            [DataMember(Name = "created_at")]
            public string CreatedAt { get; set; }
        }
    }
}

The original specification for COM is old in age, but the specification as well as the concepts are still in use, and you can still create and consume COM objects.

For .NET, you can start by looking at the following links:

Com Interop Part 1: C# Client Tutorial (C#) @ MSDN
Interoperating with unmanaged code @ MSDN

In addition, there are other specs that are very similar to COM or have semantics that are the same. Most notably, XPCOM is in use in FireFox for their plugin spec and also used internally to FireFox to connectable objects.

XPCOM @ MDN (Mozilla)

WinRT is an upcoming platform update for windows that is also heavy in COM concepts.

There are some useful here: Why is WinRT unmanaged @ StackOverflow

For .NET developers, a lot of the declarative overhead is hidden as mentioned here: WinRT demystified - Miguel de Icaza

The head of the spec is here: The Windows Runtime @ MSDN

And in the context of COM, developing WinRT components with C++ has similarities, although some syntax is borrowed from managed C++: Creating Windows Runtime Components in C++ @ MSDN

You were quite close in syntax. The key here is that d["members"] is of type Object[] / object[]. Instead of Array, you can use dynamic[] and everything works just fine.

Also note that even this declaration isn't explicitly necessary, as shown in DPeden's updated sample.

Here is the code for your updated snippet (I used a console app to test):

JavaScriptSerializer js = new JavaScriptSerializer();
dynamic d = js.Deserialize<dynamic>(json);

string key = d["key"];
string status = d["status"];
dynamic[] members = d["members"];

Console.WriteLine("key = {0}", key);
Console.WriteLine("status = {0}", status);

Console.WriteLine("members.length = {0}", members.Length);
Console.WriteLine("members type name = {0}", members.GetType().Name);
Console.WriteLine("d[\"members\"] type name = {0}", d["members"].GetType().Name);

And here is additional code showing array and member access.

Console.WriteLine("--");

for (int i = 0; i < members.Length; i++)
{
    Console.WriteLine("members[{0}][\"id\"] = {1}", i, members[i]["id"]);
    Console.WriteLine("members[{0}][\"name\"] = {1}", i, members[i]["name"]);
}

Console.WriteLine("--");

Console.WriteLine("{0}", d["members"][0]["id"]);
Console.WriteLine("{0}", d["members"][0]["name"]);

Console.ReadKey();

Added after comment:

You would need to pickup some latent detail that you can exploit to be able to accomplish that goal.

One thing that comes to mind is creating your own Forms/WPF timer at construction time and then use this and some synchronization to hide the details of coordination across threads. We can infer from your sample that construction of your poller should always happen in context of your consumer's thread.

This is a rather hack-ish way to accomplish what you want, but it can accomplish the deed because the construction of your poll-listener happens from the consumer's thread (which has a windows message pump to fuel the dispatches of Forms/WPF timers), and the rest of the operation of the class could occur from any thread as the forms Timer's tick will heartbeat from the original thread. As other comments and answers have noted, it would be best to reassess and fix the operating relationship between your polling operations and the consumer.

Here is an updated version of the class, PollingListener2 that uses a ManualResetEvent and a concealed System.Windows.Forms.Timer to ferry the polling notice across threads. Cleanup code is omitted for the sake of brevity. Requiring the use of IDisposable for explicit cleanup would be recommended in a production version of this class.

ManualResetEvent @ MSDN

public class PollingListener2
{
    System.Timers.Timer timer = new System.Timers.Timer(1000);
    public event EventHandler<EventArgs> Polled;

    System.Windows.Forms.Timer formsTimer;
    public System.Threading.ManualResetEvent pollNotice;

    public PollingListener2()
    {
        pollNotice = new System.Threading.ManualResetEvent(false);
        formsTimer = new System.Windows.Forms.Timer();
        formsTimer.Interval = 100;
        formsTimer.Tick += new EventHandler(formsTimer_Tick);
        formsTimer.Start();
        timer.Elapsed += new System.Timers.ElapsedEventHandler(PollNow);
        timer.Start();
    }

    void formsTimer_Tick(object sender, EventArgs e)
    {
        if (pollNotice.WaitOne(0))
        {
            pollNotice.Reset();
            var temp = Polled;
            if (temp != null)
            {
                Polled(this, new EventArgs());
            }
        }
    }

    void PollNow(object sender, EventArgs e)
    {
        pollNotice.Set();
    }
}

This has some precedent in the distant Win32 past where some people would use hidden windows and the like to maintain one foot in the other thread without requiring the consumer to make any significant changes to their code (sometimes no changes are necessary).

Original:

You could add a member variable on your helper class of type Control or Form and use that as the scope for a BeginInvoke() / Invoke() call on your event dispatch.

Here's a copy of your sample class, modified to behave in this manner.

public class PollingListener
{
    System.Timers.Timer timer = new System.Timers.Timer(1000);
    public event EventHandler<EventArgs> Polled;

    public PollingListener(System.Windows.Forms.Control consumer)
    {
        timer.Elapsed += new System.Timers.ElapsedEventHandler(PollNow);
        timer.Start();
        consumerContext = consumer;
    }

    System.Windows.Forms.Control consumerContext;

    void PollNow(object sender, EventArgs e)
    {
        var temp = Polled;
        if ((temp != null) && (null != consumerContext))
        {
            consumerContext.BeginInvoke(new Action(() =>
                {
                    Polled(this, new EventArgs());
                }));
        }
    }
}

Here's a sample that shows this in action. Run this in debug mode and look at your output to verify that it is working as expected.

public partial class Form1 : Form
{
    public Form1()
    {
        InitializeComponent();

        listener = new PollingListener(this);
    }

    PollingListener listener;

    private void Form1_Load(object sender, EventArgs e)
    {
        listener.Polled += new EventHandler<EventArgs>(listener_Poll);
    }

    void listener_Poll(object sender, EventArgs e)
    {
        System.Diagnostics.Debug.WriteLine("ding.");
    }
}

There are multiple ways to parse this; but in the style you've initially presented, you should try accessing it with [] indexers.

Note that your data has a root object with two properties, meta and response. Assuming response can't be null, you can access categories from it directly like this:

var root = JObject.Parse(/* your json string here */);
var categories = root["response"]["categories"];
var firstCategory = categories[0];

Note that you can use strings matching property names to descend into nested levels, and integers to index into arrays in scope.

Here is the rest of a sample program that can parse the json snippet you've provided.

using System.Windows.Forms;

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Newtonsoft.Json.Serialization;

namespace _4sqCatResponse
{
    class Program
    {
        [STAThread]
        static void Main(string[] args)
        {
            OpenFileDialog dlg = new OpenFileDialog();
            if(dlg.ShowDialog() != DialogResult.OK){return;}
            string json = System.IO.File.ReadAllText(dlg.FileName);

            var root = JObject.Parse(json);
            var categories = root["response"]["categories"];
            var firstCategory = categories[0];
            Console.WriteLine("id: {0}", firstCategory["id"]);
            Console.WriteLine("name: {0}", firstCategory["name"]);
            Console.WriteLine("pluralName: {0}", firstCategory["pluralName"]);
            Console.WriteLine("shortName: {0}", firstCategory["shortName"]);
            var icon = firstCategory["icon"];
            Console.WriteLine("icon.prefix: {0}", icon["prefix"]);
            Console.WriteLine("icon.sizes[0]: {0}", icon["sizes"][0]);
            Console.WriteLine("icon.name: {0}", icon["name"]);

            Console.ReadKey();
        }
    }
}

Also I'm a little confused by your json sample; I think you may have overlaid part of your sample or cut something out as you have categories nested in categories. If there is indeed a 2nd level categories inside your categories element and that's what you want, you can access it with this:

var categories2 = root["response"]["categories"][0]["categories"][0];
Console.WriteLine("inner categories id: {0}", categories2["id"]);

Here is the json source I used to test, copied from yours but with closing } and ] marks where needed to make it parse.

{
    "meta": {
        "code": 200
    },
    "response": {
        "categories": [{
            "id": "4d4b7104d754a06370d81259",
            "name": "Arts & Entertainment",
            "pluralName": "Arts & Entertainment",
            "shortName": "Arts & Entertainment",
            "icon": {
                "prefix": "https:\/\/foursquare.com\/img\/categories\/arts_entertainment\/default_",
                "sizes": [32, 44, 64, 88, 256],
                "name": ".png"
            },
            "categories": [{
                "id": "4bf58dd8d48988d1e1931735"
                }]
            }]
    }
}

The byte array arguments to Socket.IOControl() are specific to the IOControlCode that is specified as the first argument.

The first array is for input data, the second array is for output data.

To find more information about the specific layout for these arrays, it is helpful to start by looking at the enumeration, which lists all the possible operations for you. From there, you should cross reference with the documentation listed for the C-operable functions that Socket.IOControl() wraps. These are WSAIoctl() and ioctlsocket().

IOControlCode Enumeration @ MSDN

WSAIoctl() function @ MSDN
ioctlsocket() function @ MSDN

Per the documentation for IOControlCode.ReceiveAll:

Enable receiving all IPv4 packets on the network. The socket must have address family InterNetwork, the socket type must be Raw, and the protocol type must be IP. The current user must belong to the Administrators group on the local computer, and the socket must be bound to a specific port. This control code is supported on Windows 2000 and later operating systems. This value is equal to the Winsock 2 SIO_RCVALL constant.

Cross referencing SIO_RCVALL, we find it has an entry in the winsock documentation.

SIO_RCVALL control code @ MSDN

Reading through this entry, it mentions that the input buffer is required to select a mode of operation, with minimum size corresponding to a RCVALL_VALUE. In your statement you are passing the value 1 in the input argument array. We can check header files to see what this should mean.

You should double check this with C header files at hand, but the interface-compatible copy for wine (windows emulator) says the value for mode RCVALL_ON is 1. The .NET method should wrap the details of managing array size for you, so the code snippet you encountered is trying to enable RCVALL_ON for the socket.

Here is the link to the C header file at winehq.org: mstcpip.h at winehq.org

If you've installed C/C++ support for visual studio, you may be able to locate mstcpip.h and winsock2.h locally at a path similar to the following:

C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Include\

My copy says:

//
// Values for use with SIO_RCVALL* options
//
typedef enum {
    RCVALL_OFF             = 0,
    RCVALL_ON              = 1,
    RCVALL_SOCKETLEVELONLY = 2,
    RCVALL_IPLEVEL         = 3,
} RCVALL_VALUE, *PRCVALL_VALUE;

I couldn't find any documentation that specifically says what size byte array should be used to pass a RCVALL_VALUE, but if you look at the the samples for IOControl(), they use BitConverter.GetBytes(0) for default parameters which would have a size of 4 bytes (corresponding to a .NET int) and matches your example. This is large enough to fit a RCVALL_VALUE in C operation as well.

Both sets of types you've given work; the issue is the source data returned is an array of the core type, not a single instance. Note the square brackets in the JSON sample you've provided around the entire unit of data.

Here is a test console app that you can use to check the JSON you're seeing.

To use it, copy the JSON you have provided above into an arbitrary text file and open it from the app.

Note that deserializing as an array works fine, but single element (per your expectations) fails.

If you create a sample text file containing the same sample data but without the outer [ ] brackets, you'll see that the array deserialization fails and single elements work.

using System.Windows.Forms;
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;

namespace RptSuiteMetricsTest1
{
    class Program
    {
        public class report_suite_metrics
        {
            public string rsid { get; set; }
            public string site_title { get; set; }
            public metric[] available_metrics { get; set; }
        }

        public class metric
        {
            public string metric_name { get; set; }
            public string display_name { get; set; }
        }

        public class AvailableMetric
        {
            public string metric_name { get; set; }
            public string display_name { get; set; }
        }

        public class RootObject
        {
            public string rsid { get; set; }
            public string site_title { get; set; }
            public List<AvailableMetric> available_metrics { get; set; }
        }

        [STAThread]
        static void Main(string[] args)
        {
            OpenFileDialog dlg = new OpenFileDialog();
            DialogResult dr = dlg.ShowDialog();
            if (dr != DialogResult.OK) { return; }

            // NOTE: ...or replace this with a string containing the json data to test.
            string jsonData = System.IO.File.ReadAllText(dlg.FileName);

            Console.WriteLine("Deserialize as array");
            try
            {
                report_suite_metrics[] dataset = JsonConvert.DeserializeObject<report_suite_metrics[]>(jsonData);
                Console.WriteLine("{0} items found.", dataset.Length);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Exception of type {0}", ex.GetType().Name);
            }

            Console.WriteLine("Deserialize as array, alternate type");
            try
            {
                RootObject[] dataset = JsonConvert.DeserializeObject<RootObject[]>(jsonData);
                Console.WriteLine("{0} items found.", dataset.Length);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Exception of type {0}", ex.GetType().Name);
            }

            Console.WriteLine();

            Console.WriteLine("Deserialize as single item");
            try
            {
                report_suite_metrics dataset = JsonConvert.DeserializeObject<report_suite_metrics>(jsonData);
                Console.WriteLine("rsid=\"{0}\"", dataset.rsid);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Exception of type {0}", ex.GetType().Name);
            }

            Console.WriteLine("Deserialize as single item, alternate type");
            try
            {
                RootObject dataset = JsonConvert.DeserializeObject<RootObject>(jsonData);
                Console.WriteLine("rsid=\"{0}\"", dataset.rsid);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Exception of type {0}", ex.GetType().Name);
            }

            Console.ReadKey();
        }
    }
}

You can write a helper class that circumvents the usual handling of System.Uri and treats it purely as a string, using the Uri.OriginalString property.

Here's a code sample that does exactly that with a converter class derived from Json.Net's JsonConverter.

OriginalString Property (System.Uri) @ MSDN

One caveat is that you must update all places where you use JsonConvert to include the helper class as one of the extra JsonConverter parameters.

I've also added an example using Uri as a member variable in a class, to demonstrate that one wouldn't necessarily have to override the attributes on a class, though it might be more convenient for you. If so, you could use [JsonConverter(UriConverter)] as an attribute on members that need it.

using Newtonsoft.Json;

namespace JsonUriSerializeTest
{
    class Program
    {
        public class UriConverter : JsonConverter
        {
            public override bool CanConvert(Type objectType)
            {
                return objectType.Equals(typeof(Uri));
            }

            public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
            {
                if (reader.TokenType == JsonToken.String)
                {
                    return new Uri((string)reader.Value);
                }

                if (reader.TokenType == JsonToken.Null)
                {
                    return null;
                }

                throw new InvalidOperationException("Unhandled case for UriConverter. Check to see if this converter has been applied to the wrong serialization type.");
            }

            public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
            {
                if (null == value)
                {
                    writer.WriteNull();
                    return;
                }

                if (value is Uri)
                {
                    writer.WriteValue(((Uri)value).OriginalString);
                    return;
                }

                throw new InvalidOperationException("Unhandled case for UriConverter. Check to see if this converter has been applied to the wrong serialization type.");
            }
        }

        public class UriPair
        {
            public string label { get; set; }
            public Uri first { get; set; }
            public Uri second { get; set; }

            public void Display()
            {
                Console.WriteLine(string.Format("label:  {0}", label));
                Console.WriteLine(string.Format("first:  {0}", first));
                Console.WriteLine(string.Format("second: {0}", second));
            }
        }

        static void Main(string[] args)
        {
            string input = "http://test.com/%22foo+bar%22";
            Uri uri = new Uri(input);
            string json = JsonConvert.SerializeObject(uri, new UriConverter());
            Uri output = JsonConvert.DeserializeObject<Uri>(json, new UriConverter());

            Console.WriteLine(input);
            Console.WriteLine(output.ToString());
            Console.WriteLine();

            UriPair pair = new UriPair();
            pair.label = input;
            pair.first = null;
            pair.second = new Uri(input);
            string jsonPair = JsonConvert.SerializeObject(pair, new UriConverter());
            UriPair outputPair = JsonConvert.DeserializeObject<UriPair>(jsonPair, new UriConverter());

            outputPair.Display();
            Console.WriteLine();

            Console.ReadKey();
        }
    }
}

Updated Answer:

Looking at how JSON.Net maps XML, it takes the approach that what it sees is what it serializes, except that if it sees multiples, it will make an array. This is great for many XML DOM trees with consistent layout, but unfortunately cannot work for your purposes.

You can verify this by looking at the body for the functions SerializeGroupedNodes() and SerializeNode() in the following file source.

XmlNodeConverter.cs source code @ CodePlex, ChangeSet #63616

There's another option that I'd previously thought might be overkill but would be helpful now that we know what to expect from the default behavior on the serializing end.

Json.Net supports using custom converters derived from JsonConverter to map particular cases of values on a case-by-case basis.

We can handle this either at the serializing side or the deserializing side. I've chosen to write a solution on the deserializing side, as you probably have other existing reasons to map XML to JSON.

One great benefit is that your class can stay intact except for the override, which requires that you apply an attribute. Here's a code sample demonstrating how to use JsonAttribute and a custom converter class (MMArrayConverter) to fix your problem. Note that you will probably want to test this more thoroughly and maybe update the converter to handle other cases, say if you eventually migrate to IList<string> or some other funky case like Lazy<List<string>>, or even make it work with generics.

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Newtonsoft.Json.Converters;

namespace JsonArrayImplictConvertTest
{
    public class MMArrayConverter : JsonConverter
    {
        public override bool CanConvert(Type objectType)
        {
            return objectType.Equals(typeof(List<string>));
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            if (reader.TokenType == JsonToken.StartArray)
            {
                List<string> parseList = new List<string>();
                do
                {
                    if (reader.Read())
                    {
                        if (reader.TokenType == JsonToken.String)
                        {
                            parseList.Add((string)reader.Value);
                        }
                        else
                        {
                            if (reader.TokenType == JsonToken.Null)
                            {
                                parseList.Add(null);
                            }
                            else
                            {
                                if (reader.TokenType != JsonToken.EndArray)
                                {
                                    throw new ArgumentException(string.Format("Expected String/Null, Found JSON Token Type {0} instead", reader.TokenType.ToString()));
                                }
                            }
                        }
                    }
                    else
                    {
                        throw new InvalidOperationException("Broken JSON Input Detected");
                    }
                }
                while (reader.TokenType != JsonToken.EndArray);

                return parseList;
            }

            if (reader.TokenType == JsonToken.Null)
            {
                // TODO: You need to decide here if we want to return an empty list, or null.
                return null;
            }

            if (reader.TokenType == JsonToken.String)
            {
                List<string> singleList = new List<string>();
                singleList.Add((string)reader.Value);
                return singleList;
            }

            throw new InvalidOperationException("Unhandled case for MMArrayConverter. Check to see if this converter has been applied to the wrong serialization type.");
        }

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            // Not implemented for brevity, but you could add this if needed.
            throw new NotImplementedException();
        }
    }

    public class ModifiedXX
    {
        public string yy { get; set; }

        [JsonConverter(typeof(MMArrayConverter))]
        public List<string> mm { get; set; }

        public void Display()
        {
            Console.WriteLine("yy is {0}", this.yy);
            if (null == mm)
            {
                Console.WriteLine("mm is null");
            }
            else
            {
                Console.WriteLine("mm contains these items:");
                mm.ForEach((item) => { Console.WriteLine("  {0}", item); });
            }
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            string jsonTest1 = "{\"yy\":\"nn\", \"mm\": [ \"zzz\", \"aaa\" ] }";
            ModifiedXX obj1 = JsonConvert.DeserializeObject<ModifiedXX>(jsonTest1);
            obj1.Display();

            string jsonTest2 = "{\"yy\":\"nn\", \"mm\": \"zzz\" }";
            ModifiedXX obj2 = JsonConvert.DeserializeObject<ModifiedXX>(jsonTest2);
            obj2.Display();

            // This test is now required in case we messed up the parser state in our converter.
            string jsonTest3 = "[{\"yy\":\"nn\", \"mm\": [ \"zzz\", \"aaa\" ] },{\"yy\":\"nn\", \"mm\": \"zzz\" }]";
            List<ModifiedXX> obj3 = JsonConvert.DeserializeObject<List<ModifiedXX>>(jsonTest3);
            obj3.ForEach((obj) => { obj.Display(); });

            Console.ReadKey();
        }
    }
}

Original Answer:

It would be best to fix the JSON you're receiving at the source, as many have already pointed out. You may wish to post an update showing how the XML in your updated comment is being mapped to JSON, as that would be the best route overall.

However, if you find that this is not possible and you want some way to serialize and handle the variant value after-the-fact, you can patch things up by declaring mm to be type object, and then handling the possible cases yourself using JSON.Net's Linq support. In the two scenarios you described, you'll find that declaring mm to be type object will result in either a null, a string, or a JArray being assigned to mm by the call to DeserializeObject<>.

Here's a code sample that shows this in action. There's also a case in other circumstances where you could receive a JObject, which is also covered in this sample. Note that the member function mmAsList() does the work of patching up the difference for you. Also note that I've handled null here by returning a null for List<string>; you will probably want to revise this for your implementation.

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace JsonArrayUnionTest
{
    public class ModifiedXX
    {
        public string yy { get; set; }
        public object mm { get; set; }

        public List<string> mmAsList()
        {
            if (null == mm) { return null; }
            if (mm is JArray)
            {
                JArray mmArray = (JArray)mm;
                return mmArray.Values<string>().ToList();
            }

            if (mm is JObject)
            {
                JObject mmObj = (JObject)mm;
                if (mmObj.Type == JTokenType.String)
                {
                    return MakeList(mmObj.Value<string>());
                }
            }

            if (mm is string)
            {
                return MakeList((string)mm);
            }

            throw new ArgumentOutOfRangeException("unhandled case for serialized value for mm (cannot be converted to List<string>)");
        }

        protected List<string> MakeList(string src)
        {
            List<string> newList = new List<string>();
            newList.Add(src);
            return newList;
        }

        public void Display()
        {
            Console.WriteLine("yy is {0}", this.yy);
            List<string> mmItems = mmAsList();
            if (null == mmItems)
            {
                Console.WriteLine("mm is null");
            }
            else
            {
                Console.WriteLine("mm contains these items:");
                mmItems.ForEach((item) => { Console.WriteLine("  {0}", item); });
            }
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            string jsonTest1 = "{\"yy\":\"nn\", \"mm\": [ \"zzz\", \"aaa\" ] }";
            ModifiedXX obj1 = JsonConvert.DeserializeObject<ModifiedXX>(jsonTest1);
            obj1.Display();

            string jsonTest2 = "{\"yy\":\"nn\", \"mm\": \"zzz\" }";
            ModifiedXX obj2 = JsonConvert.DeserializeObject<ModifiedXX>(jsonTest2);
            obj2.Display();

            Console.ReadKey();
        }
    }
}

The assignment and dereference for AddRef() are correct.

When MFCreateVideoSampleFromSurface() is called, the second argument to it is the location where the pointer to the interface should be stored. You use &pSample to obtain the address to pass to the function. This matches the IMFSample ** type required. Note that the & operator on CComPtr<> via CComPtrBase<> returns the correct type.

CComPtrBase::& operator @ MSDN

ppVideoSample is type IMFSample ** as well, which requires the * operator to dereference the interface pointer. That yields a pointer of type IMFSample *, which you can call with the -> operator to access AddRef() and other functions on the interface.

Short Answer: You can get individual tokens of text via Doc.GetText("SVG"), parsing the XML for TEXT and TSPAN elements, and determining if there is layout spacing that should be treated as actual spaces. The behavior you're seeing from PDFBox is probably their attempt to make that assumption. Also, even Adobe Acrobat can return spaced text via the clipboard as PDFBox does.

Long Answer: This may cause more problems, as this may not be the original intent of the text in the PDF.

ABCpdf is doing the correct thing here, as the PDF spec only describes where things should be placed in the output medium. One can construct a PDF file that ABCpdf interprets in both styles, even though the original sentence looks nearly the same.

To demonstrate this, here is a snapshot of a document from Adobe InDesign that shows a text layout matching both cases for your sample sentence.

Note that the first row was not constructed with actual spaces, instead, the words were placed by hand in individual text regions and lined up to look approximately like a properly spaced sentence. The second row has a single sentence that has actual text spaces between the words, in a single text region.

When exported to PDF and then read in by ABCpdf, Doc.GetText("TEXT") will return the following:

ThisSentenceDoesn'tHaveAnySpacesBetweenWords.  
This Sentence Doesn't Have Any Spaces Between Words.

Thus if you wish to detect layout spaces, you must use SVG output and step through the tokens of text manually. Doc.GetText("SVG") returns text and other drawing entities as ABCpdf sees them on the page, and you can decide how you want to handle the case of layout based spacing.

You'll receive output similar to this:

<?xml version="1.0" standalone="no"?>
<svg width="612" height="792" x="0" y="0" version="1.1" baseProfile="full" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<text xml:space="preserve" x="36" y="46.1924" font-size="14" font-family="ArialMT" textLength="26.446" transform="translate(36, 46.1924) translate(-36, -46.1924)">This</text>
<text xml:space="preserve" x="66.002" y="46.1924" font-size="14" font-family="ArialMT" textLength="59.15" transform="translate(66.002, 46.1924) translate(-66.002, -46.1924)">Sentence</text>
<text xml:space="preserve" x="129.604" y="46.1924" font-size="14" font-family="ArialMT" textLength="47.46" transform="translate(129.604, 46.1924) translate(-129.604, -46.1924)">Doesn&#8217;t</text>
<text xml:space="preserve" x="181.208" y="46.1924" font-size="14" font-family="ArialMT" textLength="32.676" transform="translate(181.208, 46.1924) translate(-181.208, -46.1924)">Have</text>
<text xml:space="preserve" x="219.61" y="46.1924" font-size="14" font-family="ArialMT" textLength="24.122" transform="translate(219.61, 46.1924) translate(-219.61, -46.1924)">Any</text>
<text xml:space="preserve" x="249.612" y="46.1924" font-size="14" font-family="ArialMT" textLength="46.69" transform="translate(249.612, 46.1924) translate(-249.612, -46.1924)">Spaces</text>
<text xml:space="preserve" x="301.216" y="46.1924" font-size="14" font-family="ArialMT" textLength="54.474" transform="translate(301.216, 46.1924) translate(-301.216, -46.1924)">Between</text>
<text xml:space="preserve" x="360.016" y="46.1924" font-size="14" font-family="ArialMT" transform="translate(360.016, 46.1924) translate(-360.016, -46.1924)"><tspan textLength="13.216">W</tspan><tspan dx="-0.252" textLength="31.122">ords.</tspan></text>
<text xml:space="preserve" x="36.014" y="141.9944" font-size="14" font-family="ArialMT" transform="translate(36.014, 141.9944) translate(-36.014, -141.9944)">
<tspan textLength="181.3">This Sentence Doesn&#8217;t Have </tspan><tspan dx="-0.756" textLength="150.178">Any Spaces Between W</tspan><tspan dx="-0.252" textLength="31.122">ords.</tspan></text>
</svg>

And note that the basic structure reveals the original intent that gave you problems. (xml:space and attributes removed, whitespace modifications for the sake of example)

<?xml version="1.0" standalone="no"?>
<svg>
  <text>This</text>
  <text>Sentence</text>
  <text>Doesn&#8217;t</text>
  <text>Have</text>
  <text>Any</text>
  <text>Spaces</text>
  <text>Between</text>
  <text><tspan>W</tspan><tspan>ords.</tspan></text>
  <text>
    <tspan>This Sentence Doesn&#8217;t Have </tspan>
    <tspan>Any Spaces Between W</tspan>
    <tspan>ords.</tspan>
  </text>
</svg>

A stack imbalance occurs when the data structure used to keep track of called functions, arguments, and return values becomes corrupted or misaligned.

Most times, the stack is a memory pointer that stores the address where control will resume when the current function call exits back to the caller. There are different variants on this, sometimes the arguments to a function are also appended to the stack, as well as the return value. What is most important here is that the caller and callee should agree upon how to restore it back to the prior state when the callee exits. This agreement is frequently known as the Calling Convention.

In .NET, stack imbalances are a rare to nonexistent problem in pure managed code. However, this can be a frequent problem when calling unmanaged code, as you will need to tell the compiler how the method should be called, which then implies how the stack should be cleaned up per the calling convention.

On windows, there are a few standard calling conventions that cover the bulk of invocation cases.

stdcall - Callee will fix the stack upon exit.
fastcall - Potentially no need to fix the stack aside from return address, instead, CPU registers are used to pass arguments.
cdecl - The caller will fix the stack after the called function returns.

A formal reference is available here: Argument Passing and Naming Conventions @ MSDN

This is also of interest: X86 calling convention list @ Wikipedia

Within a given development domain, this tends not to be a problem. Each language generally has a convention that is implicit for all method calls. C/C++ uses the same convention for invocation of C/C++ calls, Python for other Python calls, etc. When crossing domains, it can become a problem if one domain doesn't use the same as another. Perhaps most common in windows, a function exported with "C" style declarations (cdecl) may cause an unbalanced stack (or worse) when called as though it had a stdcall convention, which is the method recognized by WINAPI (windows system) calls.

While your sample data is not strictly valid JSON, your attempt to parse it was pretty close.

The layout that you're seeing is sometimes used by some parties who believe that the size of their result sets could be improved (decreased) by aliasing the field names. Unfortunately it isn't as straightforward to work with this, but you can pivot the items back into objects.

My preference in these cases is to use the dynamic keyword and ExpandoObjects. You can use a class if you like, as the bulk of the work of creating an object happens in the final Select() below and you can rewrite it to map the v element sets into fields of a class instead of an ExpandoObject. The syntax to access a field is the same, as you can see by the snippet at the end that writes all the values to the Console.

Note that I've written a helper lambda to handle the case of mapping Date() into DateTime(). I'm just pointing this out as you may have a better place to put this method (an extension method on DateTime, perhaps); but there's no harm in copying and pasting it as-is into a suitable place your code.

using System.Dynamic;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;
using Newtonsoft.Json.Linq;

// ... other code removed

// You already have a means that loads your pseudo-json into results
// I used a file for the sake of this example
string results = File.ReadAllText(@"C:\temp\sample.json");

var o = JObject.Parse(results);
var headers = o.SelectToken("cols")
    .Select(x => { return new { label = x.SelectToken("label").Value<string>(), type = x.SelectToken("type").Value<string>()}; }).ToArray();
var rows = o.SelectToken("rows").Select(s => { return s.SelectToken("c");}).ToList();

Func<JConstructor, DateTime> MapAsDateTime = (s) =>
{
    // This is sloppy on my part, you should improve this as you like.
    List<int> v = new List<int>();
    foreach (JToken t in s)
    {
        v.Add(t.Value<int>());
    }
    return new DateTime(v[0], v[1], v[2], v[3], v[4], v[5]);
};

IEnumerable<dynamic> finalValues = rows.Select(s =>
    {
        var innerValues = s.ToList().Select(x => { return x.SelectToken("v"); }).ToArray();
        int i = 0;
        dynamic val = new ExpandoObject();
        IDictionary<string, object> valueMap = (IDictionary<string, object>)val;
        foreach (var innerValue in innerValues)
        {
            switch (headers[i].type)
            {
                case "string":
                    // NOTE: This can be improved, you could try to match and convert GUIDs with a regex or something else.
                    valueMap[headers[i].label] = innerValue.Value<string>();
                    break;
                case "datetime":
                    valueMap[headers[i].label] = MapAsDateTime((JConstructor)innerValue);
                    break;
                case "number":
                    // NOTE: This can be improved, your specific case needs decimal to handle things like 0.25, but many others could get by with just int
                    valueMap[headers[i].label] = innerValue.Value<decimal>();
                    break;
                case "boolean":
                    valueMap[headers[i].label] = innerValue.Value<bool>();
                    break;
                default:
                    // NOTE: You will need to add more cases if they 'define' more types.
                    throw new ArgumentException(string.Format("unhandled type \"{0}\" found in schema headers.", headers[i].type));
            }
            i++;
        }
        return val;
    });

foreach (dynamic d in finalValues)
{
    Console.WriteLine("name: {0}", d.name);
    Console.WriteLine("caller_id_number: {0}", d.caller_id_number);
    Console.WriteLine("destination_number: {0}", d.destination_number);
    Console.WriteLine("call_start: {0}", d.call_start);
    Console.WriteLine("duration: {0}", d.duration);
    Console.WriteLine("bill_seconds: {0}", d.bill_seconds);
    Console.WriteLine("uuid: {0}", d.uuid);
    Console.WriteLine("call_bill_total: {0}", d.call_bill_total);
    Console.WriteLine("recorded: {0}", d.recorded);
    Console.WriteLine("--");
}

And finally, the sample output for the very first unit of data in your sample.

name: mydomain.com
caller_id_number: 1650
destination_number: 01902321654
call_start: 6/19/2011 2:12:25 PM
duration: 3
bill_seconds: 0
uuid: 07752f6c-b203-11e0-92e6-495a2db86d6d
call_bill_total: 0
recorded: True
--

TCP is a streaming protocol, so you will always receive whatever data has arrived since the last read, up to the streaming window size on the recipient's end. This buffer can be filled with data received from multiple packets of any given size sent by the sender.

TCP Receive Window Size and Scaling @ MSDN

Although you have observed 1 unified blob of data containing 2 bytes on the receiving end in your example, it is possible to receive sequences of 1 byte by 1 byte (as sent by your sender) depending on network conditions, as well as many possible combinations of 0, 1, and 2 byte reads if you are doing non-blocking reads. When debugging on a typical uncongested LAN or a loopback setup, you will almost never see this if there is no delay on the sending side. There are ways, at lower levels of the network stack, to detect per-packet transmission but they're not used in typical TCP programming and are out of application scope.

If you switch to UDP, then every packet will be received as it was sent, which would match your expectations. This may suit you better, but keep in mind that UDP has no delivery promises and network routing may cause packets to be delivered out of order.

You should look into delimiting your data or finding some other method to detect when you have reached the end of a unit of data as defined by your application, and stick with TCP.

They're somewhat the same. LRESULT is intended to be a return value type that can hold something at least the size of a pointer on a given platform.

From Windows Data Types (Windows)

LRESULT:
Signed result of message processing.
This type is declared in WinDef.h as follows:

typedef LONG_PTR LRESULT;

LONG_PTR:
A signed long type for pointer precision. Use when casting a pointer to a long to perform pointer arithmetic.
This type is declared in <BaseTsd.h> as follows:

#if defined(_WIN64)
 typedef __int64 LONG_PTR; 
#else
 typedef long LONG_PTR;
#endif

The LRESULT type has gone through a number of changes over the years. I believe prior to Win64, it was defined as INT_PTR, but then updated for portability under Win64. A number of other types went through similar changes, such as WPARAM and LPARAM, which used to be 16 bit values in Win16, are 32 bit values in Win32, and are now 64 bit in Win64.

The actual value returned varies depending on the windows message context, as @ildjarn pointed out. There's also a chance the example source that you're reading was probably created before Win64 became a big deal, and may not have been updated or checked for portability since it was written.

For portability, the symbols are defined slightly differently depending on compile target. Hence, (INT_PTR)TRUE is better even if you know the exact value that should be returned that happens to match your build. While return 0; may return the correct result in cases where the return value is empty, this is bad form when compiling under scenarios where portability matters. Using return -1; or even return 0x7FFFFFFF; is even worse, as you may truncate, mask, or otherwise mangle the response that should be given for all platform compile targets.

Using the predefined symbols and constants is best, as the correct values and types will be substituted in for you as you change your compile targets.

Unfortunately, the behavior you're describing is not possible for Atom Tables. This is because Atom Tables in Windows are basically Hash Tables, and the mapping process handles strings in entirety and not by parts.

Of course, it almost sounds like it would be possible, as quoted from the MSDN documentation:

Applications can also use local atom tables to save time when searching for a particular string. To perform a search, an application need only place the search string in the atom table and compare the resulting atom with the atoms in the relevant structures. Comparing atoms is typically faster than comparing strings.

However, they are referring to exact matches. This limitation probably seems dated compared to what is possible with resources currently available to software. However, Atoms have been available as far back as Win16 and in those times, this facility allowed a means for applications to manage string data effectively in minimal memory. Atoms are still used now to manage window class names, and still provide decent benefits in reducing the footprint of multiple stored copies of strings.

If you need to store string data efficiently and to be able to scan by partial starting matches, a Suffix Tree is likely to meet or exceed your needs.

Looks like output parameters are not supported; while ByRef / [in, out] parameters are, but only on VARIANT parameters.

From the following KB article:

INFO: Type mismatch errors when you pass parameters from an ASP component to Visual Basic Component @ support.microsoft.com

"VBScript only supports VARIANT ByRef parameters. You can use VBScript to call a procedure that takes ByRef strings, but the default behavior of components built with Visual Basic is to fail with a type mismatch error when trying to pass ByRef parameters to these components. OLE Automation's default type-coercion function fails when asked to convert a ByRef variant into any other ByRef type."

Also, here are other links on the topic:

In, Out, In-Out, Make up your mind Already @ MSDN blogs
VBScript “Type Mismatch” issue with “[in, out] BSTR * ” parameter SO Question

It looks like you may have stumbled upon an error code (STATUS_GRAPHICS_DRIVER_MISMATCH) that is defined incorrectly in available documentation (see NTSTATUS values, shown as 0x401E0117); but it exists in the header file <ntstatus.h> in the Windows Platform SDK and matches your value.

My installed copy of the Windows SDK (v7.0, Windows 7 / .NET 3.5sp1) describes it as follows (line 13743):

//
// MessageId: STATUS_GRAPHICS_DRIVER_MISMATCH
//
// MessageText:
//
// The kernel driver detected a version mismatch between it and the user mode driver.
//
#define STATUS_GRAPHICS_DRIVER_MISMATCH  ((NTSTATUS)0xC01E0009L)

Other D3D functions have this symbol listed but not the value. Here are links to the documentation, along with the quoted text.

DxgkDdiOpenAllocation @ MSDN
STATUS_GRAPHICS_DRIVER_MISMATCH - "The display miniport driver is not compatible with the user-mode display driver that initiated the call to DxgkDdiOpenAllocation (that is, supplied private data to the display miniport driver)."

DxgkDdiCreateAllocation @ MSDN
STATUS_GRAPHICS_DRIVER_MISMATCH - "The display miniport driver is not compatible with the user-mode display driver that initiated the call to DxgkDdiCreateAllocation."

The Socket class has the Handle property, which could be used.

Socket.Handle @ MSDN

I was skeptical about whether this would work, but I was able to get it to work with no fuss at all.

To start, I made an unmanaged C++ dll to export a single function that can do something with a socket. Here's the function I created.

#include <WinSock.h>

// This is an example of an exported function.
extern "C" __declspec(dllexport) void __stdcall DoStuffWithSocket(DWORD sock)
{
  const char *data = "woot\r\n";
  send((SOCKET)sock, data, strlen(data), 0);
}

The project outputs a dll named UnmanagedSocketHandler.dll, which is the library mentioned in the P/Invoke signature in the next snippet.

Here's a quick and dirty C# console app to call that function as a Server.

using System.Net;
using System.Net.Sockets;
using System.Runtime.InteropServices;

namespace SocketHandleShareTest
{
    class Program
    {
        static void Main(string[] args)
        {
            IPEndPoint ep = new IPEndPoint(IPAddress.Any, 5353);
            Socket sListen = new Socket(AddressFamily.InterNetwork, 
                                        SocketType.Stream, ProtocolType.Tcp);
            sListen.Bind(ep);
            sListen.Listen(10);
            Socket sClient = sListen.Accept();
            Console.WriteLine("DoStuffWithSocket() enter");
            Console.ReadLine();
            DoStuffWithSocket(sClient.Handle);
            Console.WriteLine("DoStuffWithSocket() exit");
            Console.ReadLine();
            sClient.Close();
            sListen.Close();
            Console.WriteLine("Done.");
            Console.ReadLine();
        }

        [DllImport("UnmanagedSocketHandler.dll")]
        static extern void DoStuffWithSocket(IntPtr sock);
    }
}    

Last, a quick and dirty C# client app to talk to the server. I was unable to find any documentation on why this works, but it works. I'd be wary about what you try to do with the socket.

using System.Net;
using System.Net.Sockets;

namespace SocketHandleShareTestClient
{
    class Program
    {
        static void Main(string[] args)
        {
            byte[] buf = new byte[40];
            Socket s = new Socket(AddressFamily.InterNetwork,
                                  SocketType.Stream, ProtocolType.IP);
            s.Connect("localhost", 5353);
            int len = s.Receive(buf);
            Console.WriteLine("{0} bytes read.", len);
            if (len > 0)
            {
                string data = Encoding.ASCII.GetString(buf, 0, len);
                Console.WriteLine(data);
            }
            s.Close();
            Console.ReadLine();
        }
    }
}

Twitter doesn't allow basic Auth (username+password) logins through their API anymore.

You should look for a method that supports OAuth-based login.

You'll need to fetch OAuth keys for your application, which can be done from the following links. The first link allows you to enroll a new application, the second one allows you to see what applications you've registered.

New Twitter Application @ dev.twitter.com
Twitter Applications (Existing) @ dev.twitter.com

A more in-depth guide is available at the following link. You will want to read this as OAuth requires at least two steps to authenticate before you can use the twitter API.

Authenticating Requests with OAuth

There's two ways of handling them without knowing the contents at runtime.

The first way is to use JSON.Net's JObject, which will handle the case you've mentioned, plus hierarchies.

The second way is to use System.Dynamic's ExpandoObject, which is newly supported by recent releases of JSON.Net. Unfortunately it doesn't quite handle hierarchies, but JSON.Net falls back onto JObject to support them when encountered. For your purposes, it might be more straightforward.

Here's an example with the snippet and definitions you've given. Also note that ExpandoObject is directly castable to IDictionary<string, object>, whereas JObject has explicit methods to access properties.

Person corresponds to ExpandoObject, and JPerson corresponds to JObject.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using System.Dynamic;

using Newtonsoft.Json;
using Newtonsoft.Json.Converters;
using Newtonsoft.Json.Linq;

namespace YourFavoriteNamespace
{
    public class JPerson
    {
        public string FNAME;
        public string LNAME;
        public BodyType bodyType;
        public JObject belongings;
        public JObject signs;
    }

    public class Person
    {
        public string FNAME;
        public string LNAME;
        public BodyType bodyType;
        public ExpandoObject belongings;
        public ExpandoObject signs;
    }

    public class BodyType
    {
        public int height;
        public int weight;
        public string race;
        public string hair;
    }

    class Program
    {
        static void DumpDynamic(dynamic d)
        {
            IDictionary<string, object> dynMap = (IDictionary<string, object>)d;
            foreach (string key in dynMap.Keys)
            {
                Console.WriteLine("  {0}={1} (type {2})", key, dynMap[key], null == dynMap[key] ? "null" : dynMap[key].GetType().Name);
            }
        }

        static void DumpJProperties(JObject jo)
        {
            var props = jo.Properties();
            foreach (JProperty prop in props)
            {
                Console.WriteLine("  {0}={1} (type {2})", prop.Name, prop.Value, null == prop.Value ? "null" : prop.Value.GetType().Name);
            }
        }

        static void DumpPerson(Person p)
        {
            Console.WriteLine("Person");
            Console.WriteLine("  FNAME={0}", p.FNAME);
            Console.WriteLine("  LNAME={0}", p.LNAME);
            Console.WriteLine("Person.BodyType");
            Console.WriteLine("  height={0}", p.bodyType.height);
            Console.WriteLine("  weight={0}", p.bodyType.weight);
            Console.WriteLine("  race  ={0}", p.bodyType.race);
            Console.WriteLine("  hair  ={0}", p.bodyType.hair);
            Console.WriteLine("Person.belongings");
            DumpDynamic(p.belongings);
            Console.WriteLine("Person.signs");
            DumpDynamic(p.signs);
        }

        static void DumpJPerson(JPerson p)
        {
            Console.WriteLine("Person");
            Console.WriteLine("  FNAME={0}", p.FNAME);
            Console.WriteLine("  LNAME={0}", p.LNAME);
            Console.WriteLine("Person.BodyType");
            Console.WriteLine("  height={0}", p.bodyType.height);
            Console.WriteLine("  weight={0}", p.bodyType.weight);
            Console.WriteLine("  race  ={0}", p.bodyType.race);
            Console.WriteLine("  hair  ={0}", p.bodyType.hair);
            Console.WriteLine("Person.belongings");
            DumpJProperties(p.belongings);
            Console.WriteLine("Person.signs");
            DumpJProperties(p.signs);
        }

        static void DoSimplePerson()
        {
            string initJson = "{\"FNAME\":\"joe\",\"LNAME\":\"doe\",\"BodyType\":{\"height\":180,\"weight\":\"200\",\"race\":\"white\",\"hair\":\"black\"},\"BELONGINGS\":{\"shirt\":\"black\",\"Money\":15,\"randomThing\":\"anyvar\"},\"Signs\":{\"tattoo\":0,\"scar\":\"forehead\",\"glasses\":\"dorky\"}}";
            Person p = JsonConvert.DeserializeObject<Person>(initJson);
            DumpPerson(p);
            Console.ReadLine();
        }

        static void DoComplexPerson()
        {
            string initJson = "{\"FNAME\":\"joe\",\"LNAME\":\"doe\",\"BodyType\":{\"height\":180,\"weight\":\"200\",\"race\":\"white\",\"hair\":\"black\"},\"BELONGINGS\":{\"shirt\":\"black\",\"Money\":15,\"randomThing\":\"anyvar\"},\"Signs\":{\"tattoo\":0,\"scar\":\"forehead\",\"glasses\":[\"dorky\",\"hipster\"]}}";
            Person p = JsonConvert.DeserializeObject<Person>(initJson);
            DumpPerson(p);
            Console.ReadLine();
        }

        static void DoJPerson()
        {
            string initJson = "{\"FNAME\":\"joe\",\"LNAME\":\"doe\",\"BodyType\":{\"height\":180,\"weight\":\"200\",\"race\":\"white\",\"hair\":\"black\"},\"BELONGINGS\":{\"shirt\":\"black\",\"Money\":15,\"randomThing\":\"anyvar\"},\"Signs\":{\"tattoo\":0,\"scar\":\"forehead\",\"glasses\":\"dorky\"}}";
            JPerson p = JsonConvert.DeserializeObject<JPerson>(initJson);
            DumpJPerson(p);
            Console.ReadLine();
        }

        static void Main(string[] args)
        {
            DoSimplePerson();
            DoComplexPerson();
            DoJPerson();
        }
    }
}

Shared Memory is available as a data transport to and from SQL providers. You don't have to write any additional code for this if you are using built-in drivers to access your provider. Instead, you would just configure the driver and the server to use this, and your application would have to reside on the same server as your SQL provider.

The ODBC drivers available for windows support shared memory for SQL activities. To write code for these from C, you would use the ODBC API to communicate with your provider. Here's a link with a function reference.

ODBC Function Summary @ MSDN

Also note that there is support for BLOBs for all SQL providers that can handle arbitrary binary data. A list of the types known to ODBC API is available here. There's no strict requirement that your statement results must be expressible in tabular form.

SQL Data Types @ MSDN

On the other hand, if are concerned about communicating with internal SQL entities on your own terms, you might be able to patch something together via extensions to the SQL service you are using. For example, MS SQL Server allows extensions via Ole Automation Procedures or CLR (.net) Integration (available in MS SQL Server). You could potentially use these make something to communicate out-of-band. However, neither of these is easily created with a pure C solution.

Ole Automation Procedures in SQL Server @ MSDN

CLR Integration in SQL Server @ MSDN

However, I recommend that you avoid doing this, as you will find that you're at the mercy of the environment of the host service and you may not be able to participate in transactions.

If your dataset size requirements are so large that you consider RAM and direct access your best option, your needs would probably be better fulfilled by communicating only the parts that change in the dataset held outside of SQL. In addition, as a shared-memory solution is restricted to one machine, you would probably want to consider splitting the work on your dataset across multiple machines. It is more likely that you would see a performance/productivity improvement by such means than by changing how you reference data in SQL.

Last, it is tough to dictate to a SQL provider that it should avoid using filesystem storage. For MS SQL Server, one possible option is to force tempdb to reside in RAM. Here's a KB article with more details. Other DBMSs may have similar configuration options.

INF: When to use tempdb in RAM

However, the use of disk storage isn't necessarily a cause for concern. I'm unable to find a good example of how SQL providers manage a RAM / Filesystem balance, but one good analogue for SQL server is how windows is affected by page-file use. Here's a great link that details how windows behaves at high limits of operation, and how memory use doesn't necessarily correspond into overflow to disk use. Also note that applications written to run on windows are also adversely affected when the host's operation approaches these limits.

Pushing the Limits of Windows: Virtual Memory @ TechNet

If you are only doing scraping and requests, you would probably be best off using the WebRequest object that ships with .NET to do your work.

WebRequest Class @ MSDN

However, if you must have exact access to what is represented in the IE DOM, you should use Microsoft Active Accessibility to gain access. Provided you can identify the window handle or reliable location for the target IE window, and it is visible in a user session, Active Accessibility is the best way to access the target IE window and dig into the DOM. It isn't absolutely necessary to use C++, but it will probably be easier to do most of this in C++.

Active Accessibility User Interface Services @ MSDN

You'll want to use EnumChildWindows to locate (or brute force query) the DOM window either from the desktop or a frame window's handle retrieved from enumerating processes. In .NET, enumeration of processes is available from the System.Process class.

EnumChildWindows @ MSDN

EnumWindows signature @ pinvoke.net
EnumChildWindows signature @ pinvoke.net

Process.GetProcesses() @ MSDN
Process.MainWindowHandle @ MSDN

To add the type declarations you need to be able to walk the DOM in C# and to talk to MSAA, add a COM reference to 'Microsoft HTML Object Library' to your project, and add P/Invoke signatures for MSAA.

AccessibleObjectFromWindow Signature @ pinvoke.net

Once you can call MSAA, retrieve an IDispatch through Active Accessibility from the window handle. You will want to send in OBJID_NATIVEOM, which will get you an IDispatch you can interrogate.

Retrieving an IAccessible Object @ MSDN
AccessibleObjectFromWindow() @ MSDN

From here, IDispatch may be cast to IHTMLWindow2 or IHTMLDocument2 (and derivatives), which has all of the DOM script model methods and more. Unfortunately I can't remember which one is returned via this method, but in any case, IHTMLWindow2 has the document property (same as window.document in script). Either can be resolved to provide access to the DOM, which is represented by IHTMLDocument2 and all derived interfaces.

Provided your target window is following the standard method to update its caption, you can add a hook to catch the precise moment when the caption has changed.

To do this, you will need to add a windows message return hook with SetWindowsHookEx, passing it a CallWndRetProc handler from your code to receive notices upon completion of windows messages by the target window's message handler.

SetWindowsHookEx() @ MSDN
CallWndRetProc @ MSDN

The windows message you need to trap is WM_SETTEXT, which is sent to trigger the change in the default text/caption for a window.

WM_SETTEXT @ MSDN

If you would like to test this in isolation, WM_SETTEXT is an implicit message sent upon calling SetWindowText to change the window text/caption.

SetWindowText() @ MSDN

Also, you might also be able to get rid of your cbt hook, as WM_ACTIVATE, WM_MOUSEACTIVATE, and WM_SETFOCUS can be trapped with the same hook used to trap WM_SETTEXT. These messages cover various levels of activation activity and type for windows.

WM_ACTIVATE @ MSDN
WM_MOUSEACTIVATE @ MSDN
WM_SETFOCUS @ MSDN

Last, if the target window is a window you create and control in your own process, you could simply intercept those windows messages instead of using hooks. If it is impossible to use your available frameworks to intercept those messages, SetClassLongPtr is also another alternative, which may be used to implement a subclass for that type of window to allow you to intercept window messages. If you use this method, you would delegate all calls to the original windows message handler, and only act during a return from the original procedure upon receiving a message of interest.

SetClassLongPtr() @ MSDN
WindowProc @ MSDN

Database projects in Visual Studio have the features that you need. You can do schema comparisons to keep track of the differences between schema versions. They can also handle deployment, although your deployment needs may differ.

Database Projects @ MSDN

There are also other 3rd party tools that may be used as well, such as SQL Compare by Redgate.

SQL Compare by Redgate

Added:

Unfortunately, neither of these has intrinsic support for git. But, this wouldn't prevent you from bridging the gap if you don't mind doing most of the work yourself.

With Visual Studio, you can use MSBuild to automatically run scripts/executables associated with compile actions. From there, you could attach your own interactions with git.

MSBuild Reference @ MSDN

If you're looking for a tool that can automatically deploy a schema change from git, I don't know of any that exist at this time; but you could roll your own using SMO. It has all the functionality you need to be able to script SQL Server objects and execute scripts against SQL Server. You will probably have to do your own incremental scripts for table changes (as a matter of safety), but you could do drop/replace for many other objects that aren't data (Stored Procedures, User Functions, etc.).

SQL Server Management Objects @ MSDN

It sounds like PLINQ (available in .NET 4.0) has most of the parts that you would like for your solution.

Parallel LINQ @ MSDN

In particular, the place to start trying out prototypes would be the ForAll() extension method. This method can be used to perform a unit of work or task over all the items in an enumerable collection, over parallel threads.

ParallelEnumerable @ MSDN
ParallelEnumerable.ForAll() Method @ MSDN

For the flow that you're describing, you might find it useful to create an enumerator that can provide items one at a time. The C# yield return statement can be used in a method to provide an enumerable list. This can then be made into a ParallelEnumerable via the AsParallel() extension method.

yield (C#) Reference @ MSDN
ParallelEnumerable.AsParallel() @ MSDN

From there, the rest of the work is determining how to partition your transformations along the processing boundaries that you need. You can also use regular enumeration / foreach to iterate work in those places where you feel parallelism is not necessary.

You may find that you wish to tweak the number of threads used in a parallel query. The WithDegreeOfParallelism() extension method can be used to adjust how many threads are created for your work/query.

ParallelEnumerable.WithDegreeOfParallelism() @ MSDN

There may be places in your flow where you need to manipulate sets of data during parallel work. Use concurrent collections from System.Collections.Concurrent to maintain orderly access to those sets of data.

Thread-Safe Collections @ MSDN

Last, if you need even greater separation of duties, you may also wish to consider modeling and building your transformations in WF (Workflow Foundation).

Windows Workflow Foundation Intro @ MSDN

Added:

If you are positive that you don't want to use existing .net facilities for your design; you'll probably want to start modeling at a higher level of abstraction. Given that you've mentioned that you may wish to port to Java, I recommend you consider reading about the Producer::Consumer Communication model. From there, you can decide where to break up your transformation and Input/Output boundaries and work on individual parts based on lower-level primitives.

Other Producer / Consumer Questions on Stack Overflow

Also, you may wish to implement a cascade of work queues for your work flow. I don't know of very many existing solutions that are available to both .NET and java consumers; but reading up on other message queue solutions may give you some good ideas. Amazon SQS is a cloud-based one that comes to mind, and MSMQ (and derivative works) has been a solid available service on windows since Windows 2000.

Amazon Simple Queue Service @ AWS
MSMQ (Microsoft Messaging Queue) @ MSDN
Build a Queued WCF Response Service @ MSDN Magazine