Tuesday, October 30, 2012

My list of Visual Studio 2012 tweaks

In no particular order, here's a list of things that I like to do to Visual Studio 2012.
Here's one that I still need to review, but certainly looks intriguing: EntityFramework Reverse POCO Generator

Entity Framework Power Tools are worth evaluating. I haven't spent very much time with them yet, but generating a diagram from a DbContext is useful.

LINQPad isn't an extension, but is indispensable if you work with LINQ. I want to also call your attention to this excellent post on calling SQL Server stored procedures from LINQPad. I never thought that I would ever need to manipulate the results of a stored procedure call in LINQPad until today, and fortunately BillKrat's article showed me the way.

Though not part of Visual Studio, Team Foundation Server Integration Tools is incredibly useful if you need to selectively migrate portions of a project, such as Work Items only. Using the tool requires at least an intermediate level of knowledge of TFS.

Tuesday, October 23, 2012

Test Driving CouchDB and MongoDB with ExpandoObjects

I wanted to explore whether a database such as CouchDB or MongoDB was the solution to a data integration problem. (If you are unfamiliar with these databases then I highly recommend the book "NoSQL Distilled" by Martin Fowler and Pramod Sadalage.) I wanted to quickly get to the exploration phase, so didn't want to spend a lot of time- if any- developing domain models and the like for a skunkworks project like this one. The overarching goal was to expose the integrated data as easily consumable endpoints in a heterogeneous environment, but any deep discussions about service-oriented architecture were premature at this early analytic phase.

Accessing the initial seed data was easy. The source systems were Microsoft SQL Server so Entity Framework was used to extract data. ExpandoObjects came into play at this point. Commonality-variability analysis revealed a core set of fields shared between the data sources. My idea was that elements common across data sources, such as names and birth dates, would be persisted using the same set of identifiers. Elements specific to a particular source, such as maiden name or favorite color, could be picked up if needed based on a value indicating the source of the data. That's the beauty of a schemaless database- even though all of the items don't have the same 'shape' they can still all be stored together and queried.

At this point an example would probably help. Line 14 shows the use of an ExpandoObject to capture and reconcile common data elements. In other words, whether the original data was stored as 'firstname' or 'first_name' or 'fname' it is saved to the integration database using the 'FirstName' property. For the initial test phase I took the approach of testing whether the source data existed before creating a property as seen on lines 17, 19, and 21. It is quite possible that the proper approach is to create the properties regardless of whether they exist so that there is a degree of uniformity to the data.

public IEnumerable<ExpandoObject> GetPeople()
 var peopleList = new List<ExpandoObject>();

 using (var data = new sourceData.data_Entities())
  var people = from p in data.clients
      select p;

  if (people != null)
   foreach (var person in people)
    dynamic expando = new ExpandoObject();
    expando.FirstName = person.firstname;
    expando.LastName = person.lastname;
    if (!string.IsNullOrWhiteSpace(person.mi))
     expando.MiddleName = person.mi;
    if (person.birthdate.HasValue)
     expando.DOB = person.birthdate.Value;
    if (!string.IsNullOrWhiteSpace(person.gender))
     expando.Sex = person.gender;
    expando.SourceSystemId = person.clientid;
    expando.SourceSystem = "Source1";

 return peopleList.AsEnumerable();

Once the data was captured, the returned object was sent to the following method for upload to the database. In the example below, MongoDB is the destination. One nice side effect of using a list of ExpandoObjects is that this method can be reused because the input argument isn't strongly typed.

private static void UploadRecords(IEnumerable<ExpandoObject> records, string collectionName = "people")
 if (records != null)
  var stopwatch = GetAndStartStopwatch();

  MongoServer server = MongoServer.Create(connectionString);
  MongoDatabase db = server.GetDatabase("testdb");
  MongoCollection<BsonDocument> people = db.GetCollection<BsonDocument>(collectionName);
  List<BsonDocument> uploadDocs = new List<BsonDocument>();
  IEnumerable<SafeModeResult> results = null;

  foreach (var record in records)
   BsonDocument doc = new BsonDocument();
   doc.Add(record as IDictionary<string, object>);

  if (uploadDocs.Count > 0)
   results = people.InsertBatch(uploadDocs);

  log.InfoFormat("Records: {0}. Elapsed time: {1}", uploadDocs.Count, stopwatch.Elapsed);

In order to use the efficient InsertBatch method, I needed to build up an IEnumerable<BsonDocument>. A BsonDocument is "a collection of name/value pairs" and ExpandoObjects implement the IDictionary<String, Object> interface, so I could loop through the source data and add each item to a BsonDocument (see lines 13-18 above). The new BsonDocument then was added to a List<BsonDocument>, and this list then was used for the call to InsertBatch.

After inserting all the data, I was ready to start querying the database. Here's an example of a couple different items, or "documents" as they are called in MongoDB. Both of these documents are from the same MongoDB collection.

> db.people.findOne({"SourceSystem":"Source1"})
        "_id" : ObjectId("40842293c1199d1t7k6f0add"),
        "FirstName" : "JANE",
        "LastName" : "DOE",
        "MiddleName" : "X",
        "DOB" : ISODate("9999-01-01T06:00:00Z"),
        "Sex" : "FEMALE",
        "SourceSystemId" : 123,
        "SourceSystem" : "Source1"
> db.people.findOne({"SourceSystem":"Source2"})
        "_id" : ObjectId("309423b4c1299d1x7c6f155f"),
        "FirstName" : "John",
        "LastName" : "Doe",
        "SourceSystemId" : 321,
        "SourceSystem" : "Source2",
        "Sex" : "U"

So far my testing has been going well. Both CouchDB and MongoDB perform well for what I am doing. It's really easy to make sweeping changes to the database if I want to try something else by starting from scratch. Example: it's easy to delete the database if I need to make changes to my expandos. (TOO easy- Futon really needs an (optional) confirmation dialog.) The bottom line is that I saved a lot of time using ExpandoObjects. I haven't spent days crafting tailor made objects just to test some theories. While working with CouchDB I was also surprised (though I guess I shouldn't have beem) at how easily RestSharp knew what to do with ExpandoObjects. 

Next steps are to expose the data to a variety of clients and test performance. I have been using ASP.NET Web API for some time now and I can quickly get XML and JSON output rolling now that I have a populated database. I also need to focus on the variability side of things by loading the 'uncommon' fields into my test collections.

Would I recommend using this approach for a production system? That really depends on the intent of the system. By using ExpandoObjects you aren't able to take advantage of IntelliSense in Visual Studio, and while that might not be a deal breaker, it might seriously impact code maintainability.

Thursday, September 6, 2012

Integration Executive Overview

In the past I was asked to provide a brief executive overview of the state of integration efforts with a vendor's product. I felt that it was an honest, objective analysis so am sharing it here, with some parts redacted. I only had a few hours to put it together so admittedly it is a little unpolished.

Integration Lessons Learned

  1. Overview
    [Vendor] grudgingly acknowledges that ACME Corporation (ACME) has, over the course of many years, created an expansive codebase that extends the capabilities of the [Application] product. There is an opportunity cost associated with the primitive design of the integration as it stands at this time. That cost is derived from the friction of attempting to integrate with a system that is not meant to be extended.
  2. What's good?
    There are a couple of places where [Vendor] gets it right.

    • HL7 – there are a few places in [Application] that allow import and export of data in the HL7 data exchange format
    • Custom reports – [Application] allows a certain degree of customization of reports, though not all features of the toolset are available
    • Overall, the amount of integration done by [ACME] developers and consultants is impressive when you consider the simple fact that [Application] isn’t meant to be extended. Regardless, a key part of our day to day operations are based solely upon the hard work and diligence of those who had to discover how to extract information from [Application].
  3. What's bad?
    There are profound consequences to the integration choices that were made.

    • Our code is intimately connected to [Application] database, which in software development is known as ‘tight coupling’. What this means is that the slightest change [on the part of Application] can break our codebase, requiring inefficient emergency patch releases [on ACME's part] to keep everything running. Even more insidious is a change that we can’t detect because of its subtlety- think of the ‘Butterfly Effect’ here. Either way, our company is at a disadvantage because we are scrambling to resolve support issues rather than creating value.
    • Rather than a single source of truth, multiple parties over the years have hooked into the [Application] database. Each group might use and interpret the data differently. This also creates more opportunity cost because of the multiple points of failure. The Master Data Management project is a step in the right direction to unify enterprise data.
  4. What's the solution?
    Integration code should have as loose a connection as possible to its target. A metaphor to describe this is the electrical cord for a lamp. When you buy a lamp, do you connect it directly to the home wiring? Of course not- what if you want to move the lamp to a new location? We all know that there is a plug that fits a standard sized outlet. So how do we ‘plug in’ code to another party’s application?

    An Application Programming Interface (API) is a public access point for integration- the electrical outlet from our earlier lamp metaphor. It’s the way that entities can exchange information without being married to one another. Think of the current titans of the internet: Google, Facebook, Amazon, Netflix, and Twitter to name a few. Surely you have used a website or mobile app that displayed information from one of these companies. This third-party integration doesn’t occur by hooking directly into Facebook’s databases- it happens through their public API’s. (See Examples section below.)

    As your set of API’s starts to grow, you start to become concerned with enterprise architecture, which is the practice of reducing complexity and cost while increasing agility by rationally and efficiently connecting systems. There are a number of ways to approach this design, and all are beyond the scope of this document.
  5. The Future
    As new systems are purchased their vendors should be questioned as to the presence of an API. Keep in mind that an API isn’t a right- it’s a privilege. Some vendors probably don’t want customers to have access to their data because they see the opportunity for profit in costly customizations. As our ability to internally disseminate data in a more modern fashion evolves, unused features of tools that we use every day become valuable. For example, Microsoft Power Pivot allows Excel users to consume data from a variety of sources and perform high-level business intelligence operations from their desktop.
  6. Examples
    Facebook API
    Google API
    Twitter API
    Amazon API
  7. Reference
    Why You Might Need an API
    APIs: A Strategy Guide
    A Comparison of the Top Four Enterprise-Architecture Methodologies

Thursday, August 16, 2012

I Feel Like I'm Taking Crazy Pills: Ninject, SignalR, and Nuget

"Could not load file or assembly 'Ninject, Version=, Culture=neutral, PublicKeyToken=c7192dc5380945e7' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference."

After 3 hours of wrestling with this error I have solved the problem but don't understand the solution. Maybe it is so blindingly obvious that I am overlooking it. As you can see from the video link below, however, this one really confused me.

Clearly there was an outdated reference in one of the three projects that comprised my prototype ASP.NET MVC 3 solution that was using Ninject and SignalR. I checked, double-checked, triple-checked references manually as well as in NuGet and everything seemed in order.

I noticed that one of my projects had an 'orphaned' NuGet packages.config file. This file had entries for Ninject and Ninject.Extensions.Conventions, but the project didn't actually have references to these assemblies.
<?xml version="1.0" encoding="utf-8"?>
  <package id="Ninject" version="" targetFramework="net40" />
  <package id="Ninject.Extensions.Conventions" version="" targetFramework="net40" />
I like to keep my projects lean and clean so I decided to tidy up. Rather than manually deleting the orphaned file, I decided to use Nuget, thinking that this might be a more thorough approach. After NuGet was finished I tried running the solution again, though I had zero expectations that the problem was solved.

I almost fell out of my office chair when the app started up successfully.

"What in tarnation changed?" I wondered. (I didn't actually use the word 'tarnation', but hopefully you get a sense of my surprise and bewilderment.) So...to the Pending Changes window!

I was surprised to see that something had changed with the web.config file. Running a diff on the file revealed why my application had suddenly started to behave:binding redirection.

So that's it for now. Ninject and SignalR both work in harmony. I normally don't like to let mysteries like this linger unsolved, but I've got more important things to do today. (Visual Studio 2012 and Windows 8 are both available today, don't you know?)

Friday, July 20, 2012

Windows 7 dual monitor wallpaper tiles in the wrong direction

The other day I got sucked in by Lifehacker's Wallpaper Wednesday. One thing led to another and I downloaded a relaxing lake scene that would stretch across my two 23" monitors.

However, once I selected my new desktop image I noticed a problem. See the majestic trees on the left monitor? They're supposed to be on the far right of the right monitor, which is also the main display monitor. This is certainly a problem.
I futilely tried altering the monitor configuration in 'Control Panel\Appearance and Personalization\Display\Screen Resolution' but that never got the results that I wanted. Updated display drivers in case there was a software solution but that didn't solve the problem either.

I knew that my monitor-spanning image's 'Picture position' had to be set to tile in 'Control Panel\Appearance and Personalization\Personalization\Desktop Background' so I did some searching using terms such as 'tiling wrong way' and tiling 'right to left' and eventually came across a post like this one that revealed the underlying issue: Windows starts drawing the image in the top left of the primary display then wraps to the other displays.

With this knowledge I realized that I was going to have to use Paint.Net to resolve the matter.

The first step was to break up my 3840 x 1080 image into two equal distinct images. Following these steps from the Paint.Net forum I was able to produce two 1920 x 1080 images.
  1. Open original source image
  2. Click on Image > Canvas Size
  3. Uncheck Maintain aspect ratio CheckBox
  4. Set pixel size to 1920 width, leaving height as is
  5. Set Anchor to Top Left
  6. Save this half of the photo, naming 'left', '1', or whatever you choose
  7. Click undo, to undo the canvas size and repeat for the right half or reopening the source image and repeat, using 'Top Right' for step 5
Now I had to combine the two images produced above into a new 3840 x 1080 image. Again, the Paint.Net forum had the answers.
  1. Open the image that you want to appear on the left monitor first
  2. Image > Canvas Size
  3. Uncheck Maintain aspect ratio CheckBox
  4. Set Anchor to Top Right
  5. Double the width (in my case I changed 1920 to 3840)
  6. Import the image that you want to appear on the right monitor by using Layers > Import From File (by default this image's anchor is top left
  7. Save this new file
Did it work?


Clearly this solution is not applicable to all monitor configurations, but it should give you an idea of how to solve this problem.

Friday, June 1, 2012

Using a Func-y Dictionary Instead of a Switch

Recently I felt that the Strategy Pattern was appropriate for solving a problem. I thought that a switch statement would discriminate between the different strategies but I had reservations. I knew that the length of the switch statement would grow and become an eyesore. I had been using F# recently and had been really impressed with the succinctness of the language. These factors inspired me to find a more elegant solution to my problem.

The selection logic for each Concrete Strategy was simply to match a string value exposed as a property of a POCO. The result for each test was boolean, and a value of true indicated that a particular Concrete Strategy was to be used. I needed to somehow correlate this boolean test with an instantiated object. It occurred to me that a Dictionary<TKey, TValue> seemed to fulfill the need for this type of lookup. (Because I didn't want to unnecessarily instantiate objects, I threw Lazy<T> into the mix.)

Here is how the Dictionary is loaded:
Dictionary<Func<ITestedObject, bool>, Lazy<IStrategy>> lookup = new Dictionary<Func<ITestedObject, bool>, Lazy<IStrategy>>();

lookup.Add(o => o.Identifier.Equals("ABC-123", StringComparison.OrdinalIgnoreCase), new Lazy<IStrategy>(() => new ConcreteStrategy0()));
lookup.Add(o => o.Identifier.Equals("ABC-456", StringComparison.OrdinalIgnoreCase), new Lazy<IStrategy>(() => new ConcreteStrategy1()));
lookup.Add(o => o.Identifier.Equals("ABC-789", StringComparison.OrdinalIgnoreCase), new Lazy<IStrategy>(() => new ConcreteStrategy2()));

Here is how I use it. In my case I need to test whether members of an IEnumerable<ITestedObject> all have the same Identifier value. If all of the tested items have the same Identifier, which is determined by the use of the Dictionary's key as the predicate in the 'All' call, then I get the Value from the appropriate Dictionary entry. (Note that the object returned is Lazy<IStrategy>, and I'll have to get the actual Concrete Strategy through a call to Lazy<T>.Value.) Specifically:
var concreteStrategy = lookup.FirstOrDefault(d => someResult.TestedObjects.All(d.Key)).Value;

There is certainly room for refinement and improvement, but this solution worked well for me.

Friday, March 16, 2012

Mass assignment vulnerability references

I am a fan of ASP.NET MVC. I am also a fan of security. The two posts below combine those interests by describing how to mitigate mass assignment vulnerabilities. I am posting these so that my team can refer to them in future if needed.

Here's a bonus link. I can never remember this syntax: How to join two collections with LINQ

Monday, March 12, 2012

remove web transform files

Today I am sharing the process that I used to remove web.config transforms after a build. I wish that I could claim that I was able to implement this in a knowledge vacuum, but I had some help. As you can see, all I am doing is creating a list of all files not named web.config and then deleting them.

  <Target Name="MvcBuildViews" AfterTargets="AfterBuild" Condition="'$(MvcBuildViews)'=='true'">
      <DeleteAfterTransform Include="$(OutDir)_PublishedWebsites\$(TargetName)\Web.*.config" Exclude="$(OutDir)\_PublishedWebsites\$(TargetName)\Web.config" />
    <AspNetCompiler VirtualPath="temp" PhysicalPath="$(WebProjectOutputDir)" />
    <Message Text="WebProjectOutputDir: $(WebProjectOutputDir)" Importance="low" />
    <Message Text="OutDir: $(OutDir)" Importance="low" />
    <Message Text="TargetName: $(TargetName)" Importance="low" />
    <Message Text="ProjectConfigTransformFileName: $(ProjectConfigTransformFileName)" Importance="low" />
    <TransformXml Source="Web.config" Transform="$(ProjectConfigTransformFileName)" Destination="$(OutDir)\_PublishedWebsites\$(TargetName)\Web.config" />
    <Message Text="Preparing to remove config transform files." />
    <Message Text="Files: @(DeleteAfterTransform)" />
    <Delete Files="@(DeleteAfterTransform)">
      <Output TaskParameter="DeletedFiles" PropertyName="deleted" />
    <Message Text="Deleted Files: $(deleted)" />
I hope that you find this post useful.