C# && Such

Good Enough TM: OO, C#, .NET, ASP.NET, JS

[csharpandsuch]

Concurrent Collections With C#: ForEachConcurrent

Concurrent programming seems to be a theme of the day of late so I'll throw my hat in the ring.

First a disclaimer: I am not a fan of the async/await construct in C#.

In my view, async/await is a step away from the OO message-passing trend that came with LINQ, closures and collection method-chaining which I have grown to like and depend on. In the face of that async/await feels like a throwback to a procedural yesteryear I feel is best underemphasized and avoided. Not completely without merit, the construct has its place but I have yet to produce any clean implementations with it. It reminds me a lot of using VB goto labels for error handling - an untidy if acceptable way of doing something for which it seems there (at least) should be a more elegant way to express.

There are maybe one or two circumstances I have encountered where there would be a performance boost to an async task while imperative inline statements directly under execute concurrently. On many more occasions I was faced with circumstances where collections needed identical statements executed concurrently. Zealots will rightly note that AsParallel exists and is implemented well enough by the good people at Microsoft. The problem I have faced when using AsParallel on collections is that the performance benefits are far outstripped by the complications which arise with threading in general. In ASP.NET specifically, it is mostly harmful since it confuses the IIS/Katana wrapper threading model under which it runs. On top of that, AsParallel is severely limited to 8 threads and, if all that wasn't enough, there is no way to track completion of tasks requiring the code base to be sprayed with "global" variables checking for completion, locking, etc.

For all these code-smell reasons violating SOLID principles, async/await is actually a better alternative to AsParallel but the problem with async/await is that it too leads to SOLID violations (particularly the S) but in a more roundabout way. All async/await implementations force method's callees to be async, as well. Modifying existing statements to be async in any given lower level module means that all implementations of the method in higher layers would also have to be modified. This in turn begets callee modifications to be async which begets their callees to be async and so on and so on till eventually the entire architecture has been adapted into async. Bottom-up. Inadvertently.

When tackling identical concurrent statements over a collection it would be far easier if we could simply step into an async-like processing over the particular collection mid-statement then step out and continue executing all the tail end statements synchronously. Like AsParallel but without the limitations and with the simplicity of await to detect completion. Or, if you prefer, like async/await but without forcing async bubble up and with inline chaining expression of AsParallel. For this requirement, delegates, in my view, are far better suited approach than async/await construct, which I personally found cumbersome (for this purpose).

So without further delay I present ForEachConcurrent:

public static List<TD> ForEachConcurrent<T, TD>(this IEnumerable<T> src, Func<T, TD> concurrentAsync, bool isRunAsync = false, short magnitude = 2, params object[] args) { var arDataTasks = new ConcurrentStack<TD>(); List<Tuple<Func<T, TD>, T>> tasks = src .Select(thing => new Tuple<Func<T, TD>, T>(concurrentAsync, thing)) .ToList(); var states = new ConcurrentStack<IAsyncResult>(); tasks .SplitInBatchesOf(magnitude.BitMultiplesOf8()) .ForEach(batch => batch .ForEach(task => { var async = task.Item1; var arg = task.Item2; var state = async.BeginInvoke(arg, result => { using (result.AsyncWaitHandle) { arDataTasks.Push(async.EndInvoke(result)); result.AsyncWaitHandle.Close(); } }, args); //waitHandles states.Push(state); })); bool isWait = !isRunAsync; while (isWait) if (states.ToList().All(itm => itm.IsCompleted || itm.CompletedSynchronously) && arDataTasks.ToList().Count >= tasks.Count) isWait = false; //break; return arDataTasks.ToList(); }

The reason behind SplitInBatchesOf perhaps should be explained in more detail but for this post's intents and purposes it should suffice to say that: the number of concurrent operations which run on an x86/x64 machine (namely mine) is between 32 and 128. Going over that and somewhere between the CLR and the assembler the processor locks up while trying to manage the sheer volume of resources required for that number of operations to run concurrently. Running 1000 concurrent operations is completely out of the question. The best we can do is run up-to 32-128 concurrent operations (but for safety 32-64) and spawn new batches of operations to run concurrently when the previous batch is finished executing.

public static List<List<T>> SplitInBatchesOf<T>(this IEnumerable<T> source, int batchSize) { return source .SplitInBatchesOf(batchSize, itm => new List<T>(itm ?? new List<T>())) .ToList(); } static IEnumerable<TR> SplitInBatchesOf<T, TR>(this IEnumerable<T> source, int size, Func<IEnumerable<T>, TR> resultSelector = null, Func<T, TR> selector = null) { return batchElements(source, size, resultSelector ?? (itm => itm.Select(selector ?? (x => default(TR))).FirstOrDefault())); } private static IEnumerable<TR> batchElements<T, TR>(this IEnumerable<T> source, int size, Func<IEnumerable<T>, TR> resultSelector) { T[] batch = null; var count = 0; foreach (var item in source) { if (batch == null) batch = new T[size]; batch[count++] = item; if (count != size) continue; yield return resultSelector(batch.Select(x => x)); batch = null; count = 0; } if (batch != null && count > 0) yield return resultSelector(batch.Take(count)); }

So far we have succeeded in creating quite a bit of overhead to squeeze out 32-64 concurrent operations. Does all this boilerplate even have a significant performance impact? The answere is as always "it depends". In order to see how it might, we first need to take a look at some latency numbers from Norvig.

All of Norvig's original latency numbers were from 2002 so if we project them to 2015 (using the latency scale so kindly put together by Berkley's research staff) we can see that there are certain absolutes which are not going to be getting much better. Not only in the foreseeable future but ever. Specifically the roundtrip for one packet in a data center is locked at approximately 500μs. That's 500 micro-seconds or 0.5 milliseconds. Two roundtrips for one packet in the same data center will take us to 1ms hang time. The process must idle for 1ms and it must do so from here till the end of eternity. Joke. Probably not but I would hazard a guess that it won't improve within the next 20 years. It has been locked to the 0.5ms bottom since before 1995 so its a good bet it will remain there till 2035. And if it does improve it won't improve drastically by say an order of magnitude. It may very well half to 250μs but let's say that in the year 2050 we are dealing with a circumstance of sending one packet of data for a roundtrip in the same data centre four times which brings us back to 1ms = 4 * 250μs.

Keep this 1ms in mind. For our purposes we will use this number as an anchor to something concrete and rooted in real-life circumstance.

Then there are other numbers which will continue improving but not exponentially and at a decelerating rate. The reading of 1,000,000 bytes from an SSD takes approximately 200μs today and is projected to be around 100μs in two years time by the year 2017. What is significant is that this metric is on a completely different scale from network packet roundtrip times. In 2002, when these were published, to read approximately 1MB of data from SSD took over 3000μs (3ms). Five years later in 2007 that number was just under 1ms. Five years after in 2012 it was 0.3ms. In five years from then i.e. two years form now in 2017 it is projected at 0.1ms. So it is roughly improving by three times every five years. If we naively project that over the next 20 years it will be approximately be in the region of 100μs / 34 = 100μs / 81 = 1.2μs. And while we are being naive, let's assume that we can just add on time by adding on data such that if it takes 0.1ms to read 1MB of data today, it will take 1ms to read 10MB of data. And 10ms to load 100MB and so on and so forth. By that (flaky) line of reasoning in 2030 when it takes 0.0012ms to read 1MB SSD it will still take 1ms to sequentially read 1GB SSD.

Armed with this knowledge lets take 1ms as our common latency which occurs "in the wild" natively. We can always (naively) adjust the amount of data we wish to read sequentially from SSD and adjust the number of roundtrips in the data centre in order to project the same 1ms delay. If we take 1ms to be the latency we cannot avoid, what we could do to improve performance is process data concurrently where the time penalty overhead of code managing concurrent operations is nullified by 1ms latency over a number of operations e.g. over a thousand.

Let's set up an operation that does something which we can initially test for sequential performance without any latency. It will perform two branch conditional checks, a modulo arithmetic operation, a multiplication every length of collection and one return.

private static long operate(int i, int batchSize, int sleepInMs) { //http://imgur.com/usWXmhM //http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html if (sleepInMs > 0) Thread.Sleep(sleepInMs); return i % batchSize == 0 ? i * 2 : 1; }

Now that we have set that up let's run it over the size of the collection ten times, collect the performance metrics and then run that over. For this we can wrap the input and output into classes for clarity. We also might need a time measure which we can also put in a separate class.

class ToProcess { readonly short _magnitude; public ToProcess(short magnitude) { _magnitude = magnitude; } public int SleepInMs; public List<int> Items = new List<int>(); public int TotalItems { get { return Items.Count; } } public short Magnitude { get { return _magnitude; } } } class Processed { public List<List<long>> ConcTimes = new List<List<long>>(); public List<List<long>> LinearTimes = new List<List<long>>(); public bool HasLinear { get; set; } } //... static TimeMeasure _ts; //... static Processed process(ToProcess dataToProcess) { List<List<long>> logConcTimes = new List<List<long>>(); List<List<long>> logLinearTimes = new List<List<long>>(); for (int k = 0; k < 10; k++) { for (int idx = 0; idx < 10; idx++) { //linear { var range = new List<long>(); _ts = new TimeMeasure(); dataToProcess .Items .ForEach(i => range.Add(operate(i, dataToProcess.TotalItems, dataToProcess.SleepInMs)) )); excTimesLin[idx] = _ts.TimePassedInMs; } //concurrent _ts = new TimeMeasure(); var concRange = dataToProcess .Items .ForEachConcurrent(i => operate(i, dataToProcess.TotalItems, dataToProcess.SleepInMs), false, dataToProcess.Magnitude); excTimesConc[idx] = _ts.TimePassedInMs; } logConcTimes.Add(excTimesConc.ToList()); logLinearTimes.Add(excTimesLin.ToList()); } return new Processed { ConcTimes = logConcTimes, LinearTimes = logLinearTimes, HasLinear = true }; } class TimeMeasure { readonly DateTime _start = DateTime.Now; public long TimePassedInMs { get { return Convert.ToInt64(Math.Ceiling(DateTime.Now.Subtract(_start).TotalMilliseconds)); } } }

Another disclaimer: I am not a fan of "benchmarks" so I've devised an (overly) complex system to log and report execution times over a single input set many times and print average execution times. Moving along, we then calculate the average, median and mode times of running the operations 100 times in succession. Let's put it all together in a console app and see what results we can observe.

static readonly long[] excTimesLin = new long[10] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }; static readonly long[] excTimesConc = new long[10] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }; static void Main(string[] args) { ToProcess input = parseInput(args); Processed output = process(input); prompt(output); var newInput = gatherInput(); if (!string.IsNullOrEmpty(newInput)) Main(newInput.Split(new[] { ' ' })); } static ToProcess parseInput(string[] args) { int batchSize = args.Length < 1 ? 100 : int.Parse(args[0]); short magnitude = args.Length < 2 ? (short)2 : short.Parse(args[1]); var integers = new List<int>(Enumerable.Range(1, batchSize)); integers.Reverse(); var ret = new ToProcess(magnitude) { Items = integers, SleepInMs = args.Length > 2 ? int.Parse(args[2]) : 0 }; Console.WriteLine("args:{0}", ret); return ret; } static void prompt(Processed output) { if (output.HasLinear) { print(output.LinearTimes, "Linear"); Console.WriteLine(""); } print(output.ConcTimes, "Concurrent"); } static void prompt(Processed output) { if (output.HasLinear) { print(output.LinearTimes, "Linear"); Console.WriteLine(""); } print(output.ConcTimes, "Concurrent"); } static void print(List<List<long>> outputTimes, string title) { var concTime = new List<decimal>(); foreach (var concTimes in outputTimes) concTime.Add(concTimes.Mean(itm => itm)); Console.WriteLine("{0}", title); print(concTime.ToArray(), ""); Console.WriteLine(""); Console.WriteLine("{1} Ttl: >{0} ms", outputTimes.Sum(itm => itm.Sum()), title); } static void print(decimal[] excTimes, string label) { Console.Write("{0} TTE (in ms):", label.Trim()); Console.WriteLine(""); Console.Write("avg {0}, ", excTimes.Mean(itm => itm).ToString("#0.0", CultureInfo.InvariantCulture)); Console.Write("median {0}, ", excTimes.Median(itm => itm).ToString("#0.0", CultureInfo.InvariantCulture)); Console.Write("modes {"); Console.WriteLine(excTimes.Modes(itm => itm).ToXsv(", ", itm => itm.ToString("#0.0", CultureInfo.InvariantCulture)) + "}"); Console.WriteLine(""); } static string gatherInput() { Console.WriteLine(""); Console.WriteLine(""); Console.WriteLine("Run again? (cls + 'Return' clears screen; leave blank + 'Return' to quit)"); Console.Write("(space delimited) total items[, magnitude[,delay (in ms)]]>"); var inp = Console.ReadLine(); if (!string.IsNullOrEmpty(inp)) if (inp.ToLowerInvariant() == "cls") { Console.Clear(); Console.Beep(); ConsoleX.ClearConsoleHistory(); return gatherInput(); } return inp; }

Additionally we'll need extensions to calculate mean, median and average numbers over integer collections.

public static decimal Median<T>(this IEnumerable<T> xs, Func<T, decimal> f) { var ys = xs.OrderBy(x => x).Select(f).ToList(); double mid = (ys.Count - 1) / 2.0; return (ys[(int)(mid)] + ys[(int)(mid + 0.5)]) / 2; } public static decimal Mean<T>(this IEnumerable<T> list, Func<T, decimal> selector) { return list.Average(selector); } public static IEnumerable<decimal> Modes<T>(this IEnumerable<T> list, Func<T, decimal> f) { var modesList = list .GroupBy(f) .Select(grp => new { Value = grp.Key, Occurrences = grp.Count(), }) .ToList(); int maxOccurrence = modesList.Max(g => g.Occurrences); return modesList .Where(x => x.Occurrences == maxOccurrence && maxOccurrence > 1) // Thanks Rui! .Select(x => x.Value); }

Side-note: Contrary to my preference I am supplying the console application project (in Visual Studio 2013 - now available with Professional edition equivalent for free-as-in-beer) with implementation for reader to follow the rest of this post. Full code available for download via Dropbox

Running the entire Main procedure a few times to get the memory and caches "warmed up" and eliminate as much ambiguity due to loading. On the "first" run over a 1000 operations the output is predictable. Given the miniscule latencies in nanoseconds of clock cycles as well as L1 and L2 cache references, the sequential operations run two orders of magnitude better than the concurrent and batched counterparts. We can see this with the time to execute (TTE) of linear on average is 0.2ms with 0.4, 0.2 and 0.0 being the most recurring numbers (modes). This means that a good portion of the operations don't even register on the millisecond scale, probably running well under micro-seconds too. The total time to execute the same 1000 operations in 10 * 10 loop (i.e. 100,000 operations in total) is 24ms. Concurrent times are double an order of magnitude worse (on average and in total). In total it takes concurrently batched operation 1.2 seconds to execute 100,000 operations. So hands down winner - sequentially linear execution, no doubt. In fact, if we continue increasing the number of operation to 1,000,000, 10,000,000 etc. the numbers look predictably the same. Linear wins hands down every time.

(space delimited) total items[, magnitude[,delay (in ms)]]>1000 args:1000 | 2^2 * 8 => 32 | 0 Linear TTE (in ms): avg 0.2, median 0.2, modes {0.4, 0.2, 0.0} Linear Ttl: >24 ms Concurrent TTE (in ms): avg 11.9, median 11.5, modes {11.3, 11.4, 11.5} Concurrent Ttl: >1190 ms ------------------------------------------------------------------------- Run again? (cls + 'Return' clears screen; leave blank + 'Return' to quit) (space delimited) total items[, magnitude[,delay (in ms)]]>10000 args:10000 | 2^2 * 8 => 32 | 0 Linear TTE (in ms): avg 2.0, median 1.8, modes {1.7} Linear Ttl: >197 ms Concurrent TTE (in ms): avg 140.5, median 141.8, modes {} Concurrent Ttl: >14053 ms ------------------------------------------------------------------------- Run again? (cls + 'Return' clears screen; leave blank + 'Return' to quit) (space delimited) total items[, magnitude[,delay (in ms)]]>100000 args:100000 | 2^2 * 8 => 32 | 0 Linear TTE (in ms): avg 15.0, median 14.3, modes {12.7} Linear Ttl: >1504 ms Concurrent TTE (in ms): avg 1718.1, median 1701.6, modes {} Concurrent Ttl: >171814 ms

So far we have seen that this entire concurrent thing has no place when computing with local resources. To see where it does have an impact let's change up the input a bit.

Let's run over a 1000 operations only this time let's create an artificial delay in the operation of 1ms such that the thread is suspended for 1ms in addition to the time it takes to execute everything else.

(space delimited) total items[, magnitude[,delay (in ms)]]>1000 2 1 args:1000 | 2^2 * 8 => 32 | 1 Linear TTE (in ms): avg 2018.3, median 2006.2, modes {2002.0} Linear Ttl: >201831 ms Concurrent TTE (in ms): avg 265.5, median 249.0, modes {} Concurrent Ttl: >26549 ms

Quite a different picture. Just one millisecond delay in the operation over a 1000 operations and the average time for linear is 2 seconds where as concurrent is 265ms. That's an order of magnitude better for concurrent if you're keeping score. The total time to execute 10 * 10 loops of 1000 operations (i.e. 100,000 operations in total) linearly is over 3 minutes and concurrently is under half a minute. If we start increasing latency times to 2 or even 20ms we'll have to drop the number of operations to 100 because linear time takes too long for my patience.

(space delimited) total items[, magnitude[,delay (in ms)]]>100 2 2 args:100 | 2^2 * 8 => 32 | 2 Linear TTE (in ms): avg 301.1, median 301.0, modes {301.0} Linear Ttl: >30113 ms Concurrent TTE (in ms): avg 13.6, median 11.8, modes {11.6} Concurrent Ttl: >1355 ms ------------------------------------------------------------------------ Run again? (cls + 'Return' clears screen; leave blank + 'Return' to quit) (space delimited) total items[, magnitude[,delay (in ms)]]>10 2 20 args:10 | 2^2 * 8 => 32 | 20 Linear TTE (in ms): avg 211.7, median 211.2, modes {211.0} Linear Ttl: >21169 ms Concurrent TTE (in ms): avg 49.9, median 50.9, modes {} Concurrent Ttl: >4992 ms

So to answer the original question: are there any performance benefits? You would first have to identify where and why there are latencies to being with.

100 operations with a 2ms delay over 10 * 10 loops (i.e. total of 10,000 operations) takes 300ms on average and the total time is 30 seconds linearly vs 13.6ms on average and total of 1.3 seconds concurrently. Depending on the data set size that's a difference between re-architecting an entire user process to serve data in real time or schedule it to trigger sending of an email. Even for small sets of 10 operations over 10 * 10 loops (i.e. 1000 in total) a 20ms delay is a difference between 20s in total and 5s in total which frees up server resources quicker, allowing handling of ten times as many requests per second with same infrastructure.

I would not recommend implementing ForEachConcurrent upfront or in anticipation of (imagined) latencies where there probably aren't any. But post-release of initial versions; when knowledge and log data confirm an understanding that there are indeed and in fact parts of the process where connecting to remote services in the data center or on the www are taking up time which affects end-users... there and then you might want to consider this approach before rewriting entire flow or upgrading infrastructure.

Applying this technique wisely, strategically and discretely in your existing code base could save you a lot of time and perhaps money. As long as you first have a good grasp on where to apply it and a firm grip on why and how to apply it.

[csharpandsuch]

.NET Emails with Html Body via C#

Sending Html emails from an ASP.NET application need not be an elaborate fancy-footwork dance of AlternateView implementations. It can and should be straightforward one liners to literally dispatch an email message to recipients.

The .NET namespace provides, enough rope for developers to hang themselves with and hard-code static HTML string literals into the codebase for no apparent reason other than the fact that they don't know how to mix in dynamic runtime parameters with HTML output.

Separating Concerns

First and foremost the difficulty is the HTML itself which is reduced for email clients and the MS Outlook capacity to render it (as the lowest common denominator of email clients). Faced with the gigantic task, the novice is tempted by inertia to dive right in and start dealing with those HTML compatibility issues in the same code set dealing with the MailMessage constructs.

So before taking the first step in dispatching the email, step zero would be to separate out the HTML from the construct and deal only with runtime values while leaving the HTML in a static file somewhere to be edited and re-edited as the feedback about various email clients rolls in. Perhaps even create multiple static files for each client.

I typically utilize the following set of abstractions by compromising the design requirements in a back-and-forth with designers/information architects/UX specialists 'till we hit the golden ratio of broadest target of clients with a legible layout and branding mix in one HTML template. But that work is separate from the logic of MailMessage constructs and there is always if-all-else-fails room to create multiple layout templates using the approach below i.e. if your advice about less code to maintain and fail falls on deaf client/management ears.

The first thing we need is to set a convention for dyanmic runtime values embeded in the HTML layout template. I use $[identifier] with Hungarian camel case notation so that the sample HTML might be:

<img src="cid:logo_img_jpg_embeded_mime_b64_data"/><p>Hi $[strUser],</p><p>Your statement for $[strDateStamp] is below:</p><table><tr><th>Item</th><th>Qty.</th><th>Price</th></tr>$[strStatementTableRows]<tr><th></th><th>Total:</th><th>$[strTotal]</th></tr></table><p>Regards,</p><p>BrandX Team</p><p><a href="$[strSiteURL]">Visit us on our website</a></p>

This is a simple enough template so that it could be replaced relatively easily with a String.Format method statement but for demo purposes we know that there will be style attributes and many other HTML paraphernalia attached going forward which, if hard-coded in a string literal, would not be able to be previewed in any way. There is still less-than-ideal question of hard-coding '<tr><td>' in string literals and not being able to preview them but having done a few of these for various employers targeting an array of email clients I found that this is the best pragmatic compromise for the kind of coding I prefer.

After creating this HTML gem, we can store it as a file in a directory relative to the application. I'm not sure how this relates to mobile and desktop development, but for a web app we put this file in the ~/App_Data/emailers folder, and thus hard-code this relative path in the base class dealing with MailMessage constructs as follows:

public static class ServerFileSystem { public static string MapPath(params string[] paths) { return System.Web.Hosting.HostingEnvironment.MapPath(Path.Combine(paths)); } } public class SmtpEmail { private static readonly string emailerRootPath = ServerFileSystem.MapPath("~", "App_Data", "emailers"); }

So far so good. We'll add some fields and overloaded constructors and move onto the send method.

public class SmtpEmail { private static readonly string emailerRootPath = ServerFileSystem.MapPath("~", "App_Data", "emailers"); private readonly string _smtpFromAddress; private readonly string _smtpFromName; private readonly string _smtpPword; private readonly string _hostDNS; private readonly int _port; public SmtpEmail(string smtpFromAddress, string smtpFromName, string hostDNS, int port) { _smtpFromAddress = smtpFromAddress; _smtpFromName = smtpFromName; _hostDNS = hostDNS; _port = port; } public SmtpEmail(string smtpFromAddress, string smtpFromName, string smtpPword, string hostDNS, int port) { _smtpFromAddress = smtpFromAddress; _smtpFromName = smtpFromName; _smtpPword = smtpPword; _hostDNS = hostDNS; _port = port; } }

Nothing fancy here just some logic to deal with SMTP connection settings to dispatch a message:

public void Send(MailMessage message, bool isSsl) { SmtpClient client = string.IsNullOrEmpty(_smtpPword) ? new SmtpClient { Host = _hostDNS, Port = _port, EnableSsl = isSsl, DeliveryMethod = SmtpDeliveryMethod.Network, UseDefaultCredentials = false, } : new SmtpClient { Host = _hostDNS, Port = _port, EnableSsl = isSsl, DeliveryMethod = SmtpDeliveryMethod.Network, UseDefaultCredentials = false, Credentials = new NetworkCredential(_smtpFromAddress, _smtpPword), }; using (client) client.Send(message); }

Implementation of MailMessage and AlternateView 

Let's get into the code and justify the rationale after the fact:

public MailMessage Mail(AlternateView htmlBody, string address, string fullName, string subj) { return new MailMessage(new MailAddress(_smtpFromAddress, _smtpFromName), new MailAddress(address, fullName)) { Sender = new MailAddress(_smtpFromAddress, _smtpFromName), Body = "", Subject = subj, AlternateViews = { htmlBody }, IsBodyHtml = true }; }

A more-or-less a trivial abstraction to ensure the sender and the parameters go hand-in-hand.

This leaves the main issue of AlternateView, and embedding of images. To deal with those we'll need to understand the AlternateView class inner workings and embedding of images in the email. AlternateView requires that image data be mapped as resources with identifiers, not entirely unlike the src attribute of the img tag specifies where to get the image data. 

We need to get the email body HTML string representation for which I set up a file system helper class to load the data from the file:

public static class IO { public static string LoadFromStream(Stream stream) { using (stream) using (var sr = new StreamReader(stream)) { return sr.ReadToEnd(); } } }

Now we can use it in a method of the SmtpEmail class to deal with dynamic runtime values string injection into identifier placeholders:

public static string EmailBody(string which, IEnumerable<KeyValuePair<string, string>> placeHolders) { StreamReader stream = File.OpenText(Path.Combine(emailerRootPath, which)); var copy = IO.LoadFromStream(stream.BaseStream); copy = placeHolders.Aggregate(copy, (current, placeHolder) => current.Replace(string.Format("$[{0}]", placeHolder.Key), string.Format("{0}", placeHolder.Value))); return copy; }

This is hunky-dory for text only emails but AlternateView image resources now need to be added on the implementation layer.

We could try and internalize to the SmtpEmail instance but it is for my own personal preference intended as an implementation helper not the implementation itself so that logic comes in the form of overloading Html method as follows:

public static AlternateView Html(string body, IEnumerable<KeyValuePair<string, string>> images) { AlternateView html = AlternateView.CreateAlternateViewFromString(body, null, System.Net.Mime.MediaTypeNames.Text.Html); foreach (var image in images) { html.LinkedResources.Add(new LinkedResource(Path.Combine(emailerRootPath, image.Value), System.Net.Mime.MediaTypeNames.Image.Jpeg) { ContentId = image.Key }); } return html; } public static AlternateView Html(string fileName, DictionaryNullable<string, string> placeHolders, IEnumerable<KeyValuePair<string, string>> images) { return Html(EmailBody(fileName, images), placeHolders); }

OK we have constricted (constrained?) the image types to jpeg only and that is a Good EnoughTM for this purpose.

A more general approach would be to handle each image type individually but that would create overkill bloat for me so I left it out.

Also, we made the EmailBody method public for the explicit purpose of those who prefer to invoke the dynamic placeholder injection manually and do some post processing if need be and overloaded method for those who see no need for such things.

That makes the class public interface more verbose and, hence, confusing for first-time use but also more flexible for all those little just-in-caseies.

Full class definition:

public class SmtpEmail { private readonly string _smtpFromAddress; private readonly string _smtpFromName; private readonly string _smtpPword; private readonly string _hostDNS; private readonly int _port; private static readonly string emailerRootPath = ServerFileSystem.MapPath("~", "App_Data", "emailers"); public SmtpEmail(string smtpFromAddress, string smtpFromName, string hostDNS, int port) { _smtpFromAddress = smtpFromAddress; _smtpFromName = smtpFromName; _hostDNS = hostDNS; _port = port; } public SmtpEmail(string smtpFromAddress, string smtpFromName, string smtpPword, string hostDNS, int port) { _smtpFromAddress = smtpFromAddress; _smtpFromName = smtpFromName; _smtpPword = smtpPword; _hostDNS = hostDNS; _port = port; } public void Send(MailMessage message, bool isSsl) { SmtpClient client = string.IsNullOrEmpty(_smtpPword) ? new SmtpClient { Host = _hostDNS, Port = _port, EnableSsl = isSsl, DeliveryMethod = SmtpDeliveryMethod.Network, UseDefaultCredentials = false, } : new SmtpClient { Host = _hostDNS, Port = _port, EnableSsl = isSsl, DeliveryMethod = SmtpDeliveryMethod.Network, UseDefaultCredentials = false, Credentials = new NetworkCredential(_smtpFromAddress, _smtpPword), }; using (client) client.Send(message); } public static string EmailBody(string which, IEnumerable<KeyValuePair<string, string>> placeHolders) { StreamReader stream = File.OpenText(Path.Combine(emailerRootPath, which)); var copy = IO.LoadFromStream(stream.BaseStream); copy = placeHolders.Aggregate(copy, (current, placeHolder) => current.Replace(string.Format("$[{0}]", placeHolder.Key), string.Format("{0}", placeHolder.Value))); return copy; } public static AlternateView Html(string body, IEnumerable<KeyValuePair<string, string>> images) { AlternateView html = AlternateView.CreateAlternateViewFromString(body, null, System.Net.Mime.MediaTypeNames.Text.Html); foreach (var image in images) { html.LinkedResources.Add(new LinkedResource(Path.Combine(emailerRootPath, image.Value), System.Net.Mime.MediaTypeNames.Image.Jpeg) { ContentId = image.Key }); } return html; } public static AlternateView Html(string fileName, DictionaryNullable<string, string> placeHolders, IEnumerable<KeyValuePair<string, string>> images) { return Html(EmailBody(fileName, images), placeHolders); } public MailMessage Mail(AlternateView htmlBody, string address, string fullName, string subj) { return new MailMessage(new MailAddress(_smtpFromAddress, _smtpFromName), new MailAddress(address, fullName)) { Sender = new MailAddress(_smtpFromAddress, _smtpFromName), Body = "", Subject = subj, AlternateViews = { htmlBody }, IsBodyHtml = true }; } }

Putting It All Together

Before we move onto implementing a class instance, we'll create the emailers sub-directory in the App_Data sub-directory of the root directory of the web application. In it we will place a file called logo.jpg (any jpeg will do) as well as statement.tmpl.html and put the wireframe HTML layout template text from above:

<img src="cid:logo_img_jpg_embeded_mime_b64_data"/><p>Hi $[strUser],</p><p>Your statement for $[strDateStamp] is below:</p><table><tr><th>Item</th><th>Qty.</th><th>Price</th></tr>$[strStatementTableRows]<tr><th></th><th>Total:</th><th>$[strTotal]</th></tr></table><p>Regards,</p><p>BrandX Team</p><p><a href="$[strSiteURL]">Visit us on our website</a></p>

Next we'll write a static class Emailers in the web app and create an instance of the SmptEmail client:

public static class Emailers { private const bool IS_SSL = true; private static readonly SmtpEmail client = new SmtpEmail("[email protected]", "[email protected]", "mypassword", "smtp.gmail.com", 587); }

Hopefully by now it is clear that this SMTP account needs to be first created with the fine people of Google Inc over at gmail.com.

We'll implement a method to send the invoice statement as well as embedded image key-value data mappings:

public static class Emailers { private const bool IS_SSL = true; private static readonly SmtpEmail client = new SmtpEmail("[email protected]", "[email protected]", "mypassword", "smtp.gmail.com", 587); private static readonly DictionaryNullable<string, string> statementImages = new Dictionary<string, string>{ {"logo_img_jpg_embeded_mime_b64_data", "logo.jpg"} }; public static bool Statement(string userAddress, string userName, string subject, Uri site) { DictionaryNullable<string, string> dynamicVals = new DictionaryNullable<string, string> { {"strDateStamp", DateTime.Now.ToString("yyyy/MM/dd"), {"strSiteURL", site.AbsoluteUri }, {"strUser", userName} }; AlternateView body = SmtpEmail.Html("statement.tmpl.html", dynamicVals, statementImages); try { client.Send(client.Mail(body, userAddress, userName, "BrandX: " + subject), IS_SSL); return true; } catch (Exception ex) { //log or rethrow or both... } return false; } }

Missing Pieces

What is missing is statement data which will now needs to be loaded and rendered and injected as string in between <table> elements as a string representation of HTML row and cell tags.

The way I approach this is using IEnumerable<T> parameter where T is a custom object representing a statement item. Suppose we have a class Order which is the type of each item in a statement:

public class Order { public decimal Price; public string Display; public int Quantity; }

Create a method to process the orders and return corresponding HTML row string representation:

private const string TR_INVOICE_DATA = "<tr style=\"text-align: center; font-size: 10pt; height: 65px; color: #364fa2;\">"; private static string ordersHtml(string current, Order item) { return current + appendRow(item.Quantity, item.Price, item.Display); } private static string appendRow(int qnty, decimal price, string display) { return TR_STATEMENT + formatTableCell(display) + formatTableCell(qnty.ToString()) + formatTableCell((price * qnty).ToString()), "background-color: #e7e7e8;") + "</tr>"; } private static string formatTableCell(string input, string style) { return "<td style=\"" + style + "\">" + input + "</td>"; } private static string formatTableCell(string input) { return "<td>" + input + "</td>"; }

Amend the Statement emailer method to take the parameter of type IEnumerable<Order> in order to render out the individual table rows:

public static bool Statement(string userAddress, string userName, string subject, Uri site, IEnuemrable<Order> orders) { string statementHtmlRows = orders.Aggregate("", ordersHtml); DictionaryNullable<string, string> dynamicVals = new DictionaryNullable<string, string> { {"strDateStamp", DateTime.Now.ToString("yyyy/MM/dd"), {"strSiteURL", site.AbsoluteUri }, {"strUser", userName}, {"strStatementTableRows", statementHtmlRows}, {"strTotal", orders.Sum(itm => itm.Price * itm.Quantity).ToString("c")} }; AlternateView body = SmtpEmail.Html("statement.tmpl.html", dynamicVals, statementImages); try { client.Send(client.Mail(body, userAddress, userName, "BrandX: " + subject), IS_SSL); return true; } catch (Exception ex) { //log or rethrow or both... } return false; }

And finally implement this thing from elsewhere in our code as:

if(Emailers.Statement("[email protected]", "Joe Soap", "Order Statement", new Uri("http://www.brandx.com"), new List<Order>())){ //proceed }else{ //notify it didn't go through }

Loose Ends

For multiple templates we can always create additional html templates (e.g. statement.tmpl.clientY.html) and overload the Statement method if we know the destination address client.

Be aware this is probably a bad usability idea in the era of Exchange servers and multiple email clients for a single address accessed from home, web browser, mobile device and work depending on the user's circumstance.

public static bool Statement(string userAddress, string userName, string subject, Uri site, IEnuemrable<Order> orders, bool isGmail) { string statementHtmlRows = orders.Aggregate("", ordersHtml); DictionaryNullable<string, string> dynamicVals = new DictionaryNullable<string, string> { {"strDateStamp", DateTime.Now.ToString("yyyy/MM/dd"), {"strSiteURL", site.AbsoluteUri }, {"strUser", userName}, {"strStatementTableRows", statementHtmlRows}, {"strTotal", orders.Sum(itm => itm.Price * itm.Quantity).ToString("c")} }; AlternateView body = SmtpEmail.Html(isGmail ? "statement.tmpl.gmail.html" : "statement.tmpl.html", dynamicVals, statementImages); try { client.Send(client.Mail(body, userAddress, userName, "BrandX: " + subject), IS_SSL); return true; } catch (Exception ex) { //log or rethrow or both... } return false; } //... var customerAddr = string.Trim(getEmailAddressSomehow().ToLowerInvariant()); if(Emailers.Statement(customerAddr, "Joe Soap", "Order Statement", new Uri("http://www.brandx.com"), new List<Order>(), customerAddr.EndsWith("@gmail.com"))){ //proceed }else{ //notify someone it didn't go through }

In Closing Thoughts

I found this to be a straight forward way of approaching dispatiching of stylized HTML emails with a reusable class to embed images than any inline coding of HTML strings per implementation.

[csharpandsuch]

DateTime Abstractions

Trying to deal with date verbalization relative to the present in C#/.NET it became apparent I am bad at dealing with the DateTime type in the .NET namespace. I have come to realize that this is partly because I'm not astute to the date/time arithmetic but also partly because that arithmetic is not only counter-intuitive to a laymen but also counter-intuitive logically (at least, from the pure mathematical POV).

This due in part to absolute limitations like time lapses (which are planetary space-time continuum related, warts and all) and computational logic representation (bound to voltages of imperfect device transistor circuits) and in part to localization/encoding and display issues.

It is difficult to separate those things out so the .NET namespace abstractions are good enough to project a near-perfect view of the problem domain without burdening thinking about intricacies of that domain.

Until, that is, they are not by the Law of Leaky Abstractions, which then forces the logic around dates into niche cases where we are dealing with things all the way from GUI display, highlighting and stressing birthdays, appointments in planners through business logic level leap years and non-working days per UTC region all the way down to leap seconds and process wake-ups at the almost-at-the-metal levels where the road of mathematical purity meets the rubber of in-flux electronic circuitry.

Disclaimer

This post in no way attempts to solve the generic lexical parsing of copy in multiple languages. It's merely a Good EnoughTM starting point for a very specific set of phrases in (relatively) closely related languages. It is dealing with verbalization of dates relative to the present for an instance requiring localization in English, German, Afrikaans, isiZulu and isiXhosa.

All these had simple numeric rules and no gentrification of nouns to deal with, broad scope problems of linguistic localization outlined in: A Localization Horror Story 

Sepparating Concerns

To solve this particular set of date/time problems we'd like to separate out the concerns of type, values and performance issues of date/time permutations via representational units available in the computation (like ticks via integers and longs, as well as, formatted strings like UTC). Next, we'd like to put those to good use in a UI abstraction which will help deal with display and input capturing issues. Akin to MVC controller approach we'll need an end-user UI utility class which will be a view of sorts and a data interface class to act as a controller where the model will be the DateTime (and related types) in the .NET framework.

To solve particular GUI space problems via a translational wrapper (which will hold as a value a business logic date/time property) we first need an intermediate type to deal with the computational portion.

The aim is to start of small with a fairly common linguistic requirement in web applications of translating date stamps to copy indicating the relative age of an item to the current point in time, aka DateTime.Now

Nuts-n-bolts:

public class DataDate { private DateTime _value; public DateTime Value { get { return _value; } set { _value = new DateTime(value.Ticks, DateTimeKind.Local).ToUniversalTime(); } } public DateTime Local { get { return _value.ToLocalTime(); } } public DataDate() : this(DateTime.Now) { } public DataDate(DateTime dt) { Value = dt; } }

Which can be implemented internally in:

public class UIDate { public int ElapsedNowCuttoffSeconds = 1; public DataDate Data; public DateTime Value { get { return Data.Local; } } public UIDate() { } public UIDate(DateLinguistic config) { if (config == null) return; _now = config.NowP; _day = config.DayP; _week = config.WeekP; _month = config.MonthP; _year = config.YearP; _second = config.SecondP; _minute = config.MinuteP; _hour = config.HourP; _ago = config.AgoP; _days = config.DaysP; _weeks = config.WeeksP; _months = config.MonthsP; _years = config.YearsP; _seconds = config.SecondsP; _minutes = config.MinutesP; _hours = config.HoursP; _isFlipAgo = config.IsSwapAgo; } private readonly bool _isFlipAgo = false; private readonly string _day = "day"; private readonly string _week = "week"; private readonly string _month = "month"; private readonly string _year = "year"; private readonly string _now = "now"; private readonly string _second = "second"; private readonly string _minute = "minute"; private readonly string _hour = "hour"; private readonly string _ago = "ago"; private readonly string _days;// private readonly string _weeks;// = "week"; private readonly string _months;// = "month"; private readonly string _years;// = "year"; private readonly string _seconds;// = "second"; private readonly string _minutes;// = "minute"; private readonly string _hours;// = "hour"; public string Elapsed { get { TimeSpan elapsed = DateTime.Now.Subtract(Value); string word; if (elapsed.TotalDays >= 1) { double weeks = (elapsed.TotalDays / 7); if (weeks < 1) word = _days != null ? elapsed.TotalDays.Pluralize(7, _day, _week, _days) : elapsed.TotalDays.PluralizeEnglish(7, _day, _week); else { double months = (weeks / 4); double years = (months / 12); if (months < 1) word = (_weeks != null) ? weeks.Pluralize(4, _week, _month, _weeks) : weeks.PluralizeEnglish(4, _week, _month); else { if (years < 1) word = _months != null ? months.Pluralize(12, _month, _year, _months) : months.PluralizeEnglish(12, _month, _year); else word = string.Format("{0} {1}", Math.Floor(years).ToString("0"), _years != null ? years.Pluralize(_year, _years) : years.PluralizeEnglish(_year)); } } } else { if (elapsed.TotalHours >= 1) word = _hours != null ? elapsed.TotalHours.Pluralize(24, _hour, _day, _hours) : elapsed.TotalHours.PluralizeEnglish(24, _hour, _day); else { if (elapsed.TotalMinutes >= 1) word = _minutes != null ? elapsed.TotalMinutes.Pluralize(60, _minute, _hour, _minutes) : elapsed.TotalMinutes.PluralizeEnglish(60, _minute, _hour); else { word = elapsed.TotalSeconds <= ElapsedNowCuttoffSeconds ? _now : (_seconds != null ? elapsed.TotalSeconds.Pluralize(60, _second, _minute, _seconds) : elapsed.TotalSeconds.PluralizeEnglish(60, _second, _minute)); } } } return word == _now ? word : string.Format(_isFlipAgo ? "{1} {0}" : "{0} {1}", word, _ago); } } }

Using:

public class DateLinguistic { public string NowP; public string DayP; public string WeekP; public string MonthP; public string YearP; public string SecondP; public string MinuteP; public string HourP; public string AgoP; public string DaysP; public string WeeksP; public string MonthsP; public string YearsP; public string SecondsP; public string MinutesP; public string HoursP; public bool IsSwapAgo = false; }

With a little help from some extensions:

public static class Extensions { public static string Pluralize(this Double value, int factor, string label, string factorlabel, string prularizer) { return (value / factor) > 1 ? String.Format("{0} {1}", Math.Floor(value / factor).ToString("0"), Pluralize((value / factor), factorlabel, prularizer)) : String.Format("{0} {1}", Math.Floor(value).ToString("0"), Pluralize(value, label, prularizer)); } public static string PluralizeEnglish(this Double value, int factor, string label, string factorlabel) { return (value / factor) > 1 ? String.Format("{0} {1}", Math.Floor(value / factor).ToString("0"), PluralizeEnglish((value / factor), factorlabel)) : String.Format("{0} {1}", Math.Floor(value).ToString("0"), PluralizeEnglish(value, label)); } public static string PluralizeEnglish(this Double rounded, string word) { return rounded.Pluralize(word, word + "s"); //: (rounded <= 1) ? word : word + "+"; } public static string Pluralize(this Double rounded, string word, string prularizer) { return rounded >= 2 ? prularizer : word;// (rounded <= 1) ? word : string.Format("{0}(+)", word); } }

Simple Demonstration

var pirate = new UIDateDateLinguistic { IsSwapAgo = true, AgoP = "this be before", DayP = "sunrise", DaysP = "sunrises", HourP = "knot", HoursP = "knots", NowP = "argh!", MinuteP = "aft", MinutesP = "afts", MonthP = "moon", MonthsP = "moons", SecondP = "loft", SecondsP = "lofts", WeekP = "fortnight", WeeksP = "fortnights", YearP = "yargh", YearsP = "yarghs" }; UIDateDateLinguistic english = null; foreach (var lingo in new List<UIDate.DateLinguistic> { pirate, english }) { var daterizer = new UIDate(lingo) { Data = new DataDate(DateTime.Now), ElapsedNowCuttoffSeconds = 16 }; Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 0, 0, 5))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 0, 0, 15))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 0, 0, 29))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 0, 0, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 0, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 0, 2, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 0, 3, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 0, 35, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(0, 12, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(1, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(2, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(5, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(8, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(18, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(28, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(48, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(94, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(335, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(384, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); daterizer.Data = new DataDate(DateTime.Now.Subtract(new TimeSpan(1384, 1, 1, 30))); Console.WriteLine(daterizer.Elapsed); } // argh! // argh! // argh! // this be before 29 lofts // this be before 30 lofts // this be before 1 aft // this be before 2 afts // this be before 3 afts // this be before 35 afts // this be before 1 knot // this be before 12 knots // this be before 1 sunrise // this be before 2 sunrises // this be before 5 sunrises // this be before 1 fortnight // this be before 2 fortnights // this be before 1 moon // this be before 1 moon // this be before 3 moons // this be before 11 moons // this be before 1 yargh // this be before 4 yarghs // now // now // now // 29 seconds ago // 30 seconds ago // 1 minute ago // 2 minutes ago // 3 minutes ago // 35 minutes ago // 1 hour ago // 12 hours ago // 1 day ago // 2 days ago // 5 days ago // 1 week ago // 2 weeks ago // 1 month ago // 1 month ago // 3 months ago // 11 months ago // 1 year ago // 4 years ago

Bonus Helpers

Some simple extensions to DateTime to make date comparisons more legible.

public static class DateTimeExtensions { public static bool IsAfterDate(this DateTime dt, DateTime compareTo) { return dt.StartOfDay().IsAfter(compareTo.StartOfDay()); } public static bool IsBeforeDate(this DateTime dt, DateTime compareTo) { return dt.StartOfDay().IsBefore(compareTo.StartOfDay()); } public static bool IsBefore(this DateTime dt, DateTime compareTo) { return !dt.IsAfter(compareTo); } public static bool IsAfter(this DateTime dt, DateTime compareTo) { return dt.Subtract(compareTo).TotalSeconds >= 0; } public static bool IsInRange(this DateTime dt, DateTime start, DateTime end) { return dt.IsBefore(end) && dt.IsAfter(start); } }

[csharpandsuch]

Money Helper Class

Here's a little utility class to help with currency arithmetic and permutation. Decimal is generic enough to accomplish all this but I like to keep business rules logic encapsulated so I wrapped all the type value conversions into one class. Hope you find it useful (nothing further to add; should be self-explanatory)

:

decimal? tester = null; Console.WriteLine("value={0};", tester); //value=; Console.WriteLine("Money={0};", (Money)tester); //Money=[your_currency_symbol] 0[your_separator]00; tester = 0; Console.WriteLine("value={0};", tester); //value=0; Console.WriteLine("Money={0};", (Money)tester); //Money=[your_currency_symbol] 0[your_separator]00; double dblTester = 12.45; Console.WriteLine("value={0};", dblTester); //value=12.45; Console.WriteLine("Money={0};", (Money)dblTester); //Money=[your_currency_symbol] 12[your_separator]45; //... public class Money { private readonly decimal _value; #region constructors public Money() { } public Money(string val) : this() { decimal tmp; _value = decimal.TryParse(val, out tmp) ? tmp : 0; } public Money(decimal val) : this() { _value = val; } public Money(double val) : this() { _value = Decimal.Parse(val.ToString(CultureInfo.InvariantCulture)); } public Money(int valInCents) : this((decimal)((decimal)valInCents) / 100) { } public Money(long valInCents) : this((decimal)((decimal)valInCents) / 100) { } #endregion public static explicit operator Money(decimal? actual) { return (Money)(actual ?? 0); } public static explicit operator Money(double actual) { return new Money(actual); } public static explicit operator Money(double? actual) { return new Money(actual ?? 0); } public static explicit operator Money(decimal actual) { return new Money(actual); } public static implicit operator decimal(Money actual) { return actual._value; } public static explicit operator long(Money actual) { return (long)(actual._value * 100); } public override string ToString() { return _value.ToString("c", CultureInfo.CurrentCulture); } public string ToString(CultureInfo format) { return _value.ToString("c", format); } public string ToString(bool is_raw) { return is_raw ? _value.ToString("0.00") : ToString(); } public long Round { get { return (long)_value; } } public long Cents { get { return ((long)((_value - Round) * 100)); } } }

[csharpandsuch]

Y Combinator With Lambda Expressions

Please note that this post implies a working knowledge of C# lambda expressions and its intended purposes is helping illustrate Y Combinators to those already familiar with C# lambda expressions, not demonstrating how C# lambda expressions work. The code at the bottom is both an explanation and the purpose.

Some Background

The biggest motivation for lambda calculus is that it provides simple semantics for computation, enabling properties of computation to be studied formally. Lambda calculus (λ-calculus) incorporates 2 simplifications which make these semantics simple.

The first simplification is to treat functions "anonymously", without giving them explicit names.

The second is that to only use functions of a single input.

The solution to creating functions for multiple arguments is Currying (aptly named after Haskell Curry who also borrows his first name to the famous programming language). Functions are not given a name, so it's impossible to recurse by using the name. The solution is the Y-Combinator.

Haskell Curry and Alan Turing both came up with ways to write lambda expressions with the same self application and replication. Their version however, also replicates the application of a given function. These have come to be known as Y-Combinators in the untyped lambda calculus and are defined as follows:

Y = λf·(λx·f (x x)) (λx·f (x x))

One of deepest mysteries in the functional programming: the Y-Combinator.

In C#, lambda expressions have been implemented, precisely to allow mapping of higher-order functions which means that we can delegate the implementation of a general purpose Y combinator in pure C# with lambda expressions.

So without further rambling: I give you a slow-yet-pointless Y combinator with C# lambda expressions.

public delegate T YCombDelegate<T>(YCombDelegate<T> self); public delegate T Expression<T>(T function); public delegate T Evaluator<T>(Expression<T> higherFunction); public static class Y<T> { private static readonly YCombDelegate<Evaluator<Func<T, T>>> YCombinator = f => expr => x => expr(f(f)(expr))(x); public static readonly Evaluator<Func<T, T>> Map = YCombinator(YCombinator); }

Although mostly useless in Real WorldTM software applications, understanding how the Y-Combinator works and why it is important gives the programmer an important insight into the nature of functional programming. The main benefit of this whole exercise is feeling good about understanding the Y combinator.

To see an implementation here are some well known examples of the Fibonacci sequence and finding a factorial in pure C#:

public class MathUtil { public static int Fibonacci(int nthNumber) { return Y<int>.Map( f => x => x >= 3 ? f(x - 1) + f(x - 2) : 1) (nthNumber); } public static int Factorial(int nthNumber) { return Y<int>.Map( f => x => x >= 2 ? x * f(x - 1) : 1) (nthNumber); } }

Such that:

Console.WriteLine(MathUtil.Fibonacci(10)); //55 Console.WriteLine(MathUtil.Factorial(10)); //3628800

[csharpandsuch]

Query String Params Collection

Uri and UriBuilder are useful utilities for constructing valid HTTP URL query string parameters. In the web domain of discourse I find I'm hinging a lot of logic on query strings which I am forced to deal with as strings. This is a pointless conversion from Request.QueryString since they are clear cut collections of key/value pairs with clear cut standard encoding rules. So I made a wrapper to extend the SerializableDictionary class from a previous post entry, which will represent the collection of query parameters to append to, modify and extract from a valid URL string.

We'll first extend SerializableDictionary class and make it "nullable" by overriding the [] operator as with DictionaryNullable logic. This may or may not cause performance issues but we should never be dealing with sets of URL query parameters longer than 10-12 items (depending on the hard character limit of the HTTP header to which the URL string contributes - off the top of my head that should be at maximum 255 characters but you can verify that on wikipedia). Such a small collection should not add more than a few milliseconds to parse and reference using the indexing override

public class UriQry : SerializableDictionary<string, string> { public UriQry() { } public UriQry(string name, string value) { this[name] = value; } public new string this[string key] { get { if (ContainsKey(key)) return base[key]; return null; } set { if (ContainsKey(key)) base[key] = value; else Add(key, value); } } }

The next thing we want to add is an easy way to convert an object instance of UriQry type to and from URL string values.

public override string ToString() { return Keys .Aggregate(string.Empty, (current, key) => current + String.Format("{0}={1}&", HttpUtility.UrlEncode(key, Encoding.ASCII), HttpUtility.UrlEncode(this[key], Encoding.ASCII))) .TrimEnd('&'); } public static UriQry FromString(string qStr) { var obj = new UriQry(); if (!string.IsNullOrEmpty(qStr)) { string str = HttpUtility.UrlDecode(qStr, Encoding.ASCII).Replace("&", "&").TrimStart(new char[] { '?' }); foreach (string keyVal in str.Split(new char[] { '&' })) { string[] pair = keyVal.Split(new char[] { '=' }); if (pair.Length <= 0) { obj[keyVal] = ""; } else if (pair.Length <= 1) { obj[pair[0]] = ""; } else { string val = ""; for (int i = 1; i < pair.Length; i++) { val = string.Concat(val, pair[i]); } obj[pair[0]] = val; } } } return obj; }

Equipped with the above-defined class we can set up a trivial use case:

var uriQry = new UriQry{{"paramName", "paramValue"}}; Console.WriteLine(uriQry); //paramName=paramValue uriQry = UriQry.FromString("?q=buy+stuff&sId=08130198230198239&thing=Hello World!!!"); Console.WriteLine(uriQry); //q=buy+stuff&sId=08130198230198239&thing=Hello+World!!!

OK so far so good. But the real usefulness would come in having the ability to combine existing Uri and UriBuilder to take advantage of this utility.

So we'll add a Combine method to the UriQry class to concatenate 2 sets of UriQry collections and return a UriQry instance with a combined collection. I'm sure there's a cleaner way to do this by extending IEnumerable instead of Dictionary<string,string> but this is just my way of skinning the cat:

public UriQry Combine(UriQry qPairs) { UriQry ret = new UriQry(); foreach (KeyValuePair<string, string> keyValuePair in qPairs) { ret[keyValuePair.Key] = keyValuePair.Value; } return Convert(ret.ToString(), true); } public UriQry Convert(string qStr) { return Convert(qStr, false); } public UriQry Convert(string qStr, bool append) { if (!append) Clear(); UriQry tmp = FromString(qStr); foreach (KeyValuePair<string, string> pair in tmp) { this[pair.Key] = pair.Value; } return this; }

Additionally, we are going to extend Uri and UriBuilder classes with utility methods to append query string params:

public static class UriBuilderExtensions { public static UriBuilder AppendQuery(this UriBuilder uriBuilder, Dictionary<string, string> paramss) { uriBuilder.Query = UriQry .FromString(uriBuilder.Query) .Combine(new UriQry { paramss }) .ToString(); return uriBuilder; } } public static class UriExtensions { public static Uri AppendQuery(this Uri uri, Dictionary<string, string> paramss) { return new UriBuilder(uri).AppendQuery(paramss).Uri; } }

In a similar vein, we can implement a Combine method as a Uri extension to take advantage of the extensions:

public static class UriExtensions { public static Uri AppendQuery(this Uri uri, Dictionary<string, string> paramss) { return new UriBuilder(uri).AppendQuery(paramss).Uri; } public static Uri Combine(this Uri absBaseUri, params string[] relativeUris) { Uri bUri = new Uri(absBaseUri.PathAndQuery, UriKind.Relative); var ret = relativeUris.Aggregate(bUri, (current, relativeUri) => current.Combine(new Uri(relativeUri, UriKind.Relative))); return new Uri(absBaseUri, ret); } public static Uri Combine(this Uri relativeBaseUri, Uri relativeUri) { if (relativeBaseUri == null) { throw new ArgumentNullException("relativeBaseUri"); } if (relativeUri == null) { throw new ArgumentNullException("relativeUri"); } string baseUrl = VirtualPathUtility.AppendTrailingSlash(relativeBaseUri.ToString()); string combinedUrl = VirtualPathUtility.Combine(baseUrl, relativeUri.ToString()); return new Uri(combinedUrl, UriKind.Relative); } }

Putting It To Good Use

Here's a Global.asax implementation intercepting the application authentication request event in one (albeit very long) line.

protected void Application_AuthenticateRequest(object sender, EventArgs e) { if (Request.IsPostBack() && !Request.IsAuthentication()) { if (!Request.IsAuthenticated) { var qry = new UriQry { { "isLoginExpired", "yes" } }; var uri = new Uri("http://www.ourSite.com/aspDotNetApp/"); Response.Redirect(uri.Combine(FormsAuthentication.LoginUrl).AppendQuery(qry).PathAndQuery, true); } } }

Another example is with a requirement to preserve and clone query string parameters from an arbitrary request:

Uri relUri; //... //by virtue of getting data from somewhere //the equivalent of this happens: relUri = new Uri("/some/path/?a=shit&ton=of&dirty=data&params=4", UriKind.Relative); //... UriBuilder uriBuilder = new UriBuilder(new Uri(new Uri("http://domain.com"), relUri)); uriBuilder.Query = UriQry.FromString(uriBuilder.Query).Combine(new UriQry { { "username", "BobRoberts" }, { "password", "01234" } }).ToString(); Console.WriteLine(uriBuilder.Uri.PathAndQuery);

Another straightforward example making code more legible (in combination with HttpRemoteCall class from another post):

int postType; string dataId; string userId; bool isComment; //... var stemUrl = new Uri("http://domain.com/").Combine("/app/approve.ashx", true); UriQry qry = new UriQry { new Dictionary<string, string> { {"type", postType.ToString()}, {"id", dataId}, {"usr", userId}, {"comment", isComment ? "yes" : "no"}, } }; var webReq = new HttpRemoteCall(stemUrl.AppendQuery(qry).AbsoluteUri); webReq.Create(); var webResponse = webReq.Response();

Complete code:

using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Util; using System.Web; public class UriQry : SerializableDictionary<string, string> { public UriQry() { } public UriQry(string name, string value) { this[name] = value; } public new string this[string key] { get { if (ContainsKey(key)) return base[key]; return null; } set { if (ContainsKey(key)) base[key] = value; else Add(key, value); } } public override string ToString() { return Keys .Aggregate(string.Empty, (current, key) => current + String.Format("{0}={1}&", HttpUtility.UrlEncode(key, Encoding.ASCII), HttpUtility.UrlEncode(this[key], Encoding.ASCII))) .TrimEnd('&'); } public UriQry Combine(UriQry qPairs) { UriQry ret = new UriQry(); foreach (KeyValuePair<string, string> keyValuePair in qPairs) { ret[keyValuePair.Key] = keyValuePair.Value; } return Convert(ret.ToString(), true); } public UriQry Convert(string qStr) { return Convert(qStr, false); } public UriQry Convert(string qStr, bool append) { if (!append) Clear(); UriQry tmp = FromString(qStr); foreach (KeyValuePair<string, string> pair in tmp) { this[pair.Key] = pair.Value; } return this; } public static UriQry FromString(string qStr) { var obj = new UriQry(); if (!string.IsNullOrEmpty(qStr)) { string str = HttpUtility.UrlDecode(qStr, Encoding.ASCII).Replace("&", "&").TrimStart(new char[] { '?' }); foreach (string keyVal in str.Split(new char[] { '&' })) { string[] pair = keyVal.Split(new char[] { '=' }); if (pair.Length <= 0) { obj[keyVal] = ""; } else if (pair.Length <= 1) { obj[pair[0]] = ""; } else { string val = ""; for (int i = 1; i < pair.Length; i++) { val = string.Concat(val, pair[i]); } obj[pair[0]] = val; } } } return obj; } } public static class UriBuilderExtensions { public static UriBuilder AppendQuery(this UriBuilder uriBuilder, Dictionary<string, string> paramss) { uriBuilder.Query = UriQry .FromString(uriBuilder.Query) .Combine(new UriQry { paramss }) .ToString(); return uriBuilder; } } public static class UriExtensions { public static Uri AppendQuery(this Uri uri, Dictionary<string, string> paramss) { return new UriBuilder(uri).AppendQuery(paramss).Uri; } public static Uri Combine(this Uri absBaseUri, params string[] relativeUris) { Uri bUri = new Uri(absBaseUri.PathAndQuery, UriKind.Relative); var ret = relativeUris.Aggregate(bUri, (current, relativeUri) => current.Combine(new Uri(relativeUri, UriKind.Relative))); return new Uri(absBaseUri, ret); } public static Uri Combine(this Uri relativeBaseUri, Uri relativeUri) { if (relativeBaseUri == null) { throw new ArgumentNullException("relativeBaseUri"); } if (relativeUri == null) { throw new ArgumentNullException("relativeUri"); } string baseUrl = VirtualPathUtility.AppendTrailingSlash(relativeBaseUri.ToString()); string combinedUrl = VirtualPathUtility.Combine(baseUrl, relativeUri.ToString()); return new Uri(combinedUrl, UriKind.Relative); } }

[csharpandsuch]

Indexed Dictionary

I wanted to reference data in a dictionary based on the index order of the key without having to know the key value. So I extended Dictionary<TK,T> into DictionaryIndexable<TK, T> class which allows retrieval of values from a dictionary based on the order of keys.

Disclaimer: Please note this is not intended for general purpose use the integer type makes it too brittle for a general purpose indexable dictionary. More work and review is required to get it to that point. For instance, keys should be sortable, thus IComparable<TK> should be a generic constraint. As it is, it's broken since key sort order is undefined in the general.

You might want to consider KeyCollection if you are looking for a general purpose mechanism.

The benefits of using DictionaryIndexable instead of Dictionary is in those few and far circumstances which I've encountered more often than not in BlahBlahCollection of an ASP.NET control, where on event binding we know the index of the selected item but not the value AND it is handy to hold the control-bound data in a structure which is... well a Dictionary (which, by inheritance, DictionaryIndexable is).

It all hinges on a KeyOfIndex method which returns the key value from the Keys collection at runtime based on the index.

public class DictionaryIndexable<TK, T> : Dictionary<TK, T> { public TK KeyOfIndex(int idx) { return this.ElementAt(idx).Key;//return Keys.Skip(idx).First(); } }

Using it we can get the indexed value from a dictionary, bypassing the circumstance where the key type is integer:

public T this[int idx] { get { if (typeof(TK) == typeof(int)) { var key = (TK)Convert.ChangeType(idx, typeof(TK)); return base[key]; } return this[KeyOfIndex(idx)]; } set { if (typeof (TK) != typeof (int)) this[KeyOfIndex(idx)] = value; else { var key = (TK) Convert.ChangeType(idx, typeof (TK)); base[key] = value; } } }

What is left is to implement the IDictionary interface in order to extend the class from the Dictionary class.

Complete class:

using System.Collections.Generic; using System.Linq; using System.Runtime.Serialization; namespace System.Util { public class DictionaryIndexable<TK, T> : Dictionary<TK, T> { public static DictionaryIndexable<TK, T> FromDictionary(Dictionary<TK, T> actual) { return new DictionaryIndexable<TK, T>(actual); } public Dictionary<TK, T> ToDictionary() { return new Dictionary<TK, T>(this); } public DictionaryIndexable() : base() { } public DictionaryIndexable(SerializationInfo info, StreamingContext context) : base(info, context) { } public DictionaryIndexable(int capacity) : base(capacity) { } public DictionaryIndexable(int capacity, IEqualityComparer<TK> iEq) : base(capacity, iEq) { } public DictionaryIndexable(IDictionary<TK, T> iDic, IEqualityComparer<TK> iEq) : base(iDic, iEq) { } public DictionaryIndexable(IDictionary<TK, T> iDic) : base(iDic) { } public TK KeyOfIndex(int idx) { return this.ElementAt(idx).Key;//return Keys.Skip(idx).First(); } public T this[int idx] { get { if (typeof(TK) == typeof(int)) { var key = (TK)Convert.ChangeType(idx, typeof(TK)); return base[key]; } return this[KeyOfIndex(idx)]; } set { if (typeof (TK) != typeof (int)) this[KeyOfIndex(idx)] = value; else { var key = (TK) Convert.ChangeType(idx, typeof (TK)); base[key] = value; } } } } }

[csharpandsuch]

ASP.NET Image Handling

In ASP.NET web development (weather it be WebForms, MVC, MFP, WCT, WFP, DMV, KGB, BMW or even Silverlight) I have come across a need to dynamically scale images and/or serve the image data to the browser via a dynamic request handler.

I have found the Drawing namespace for the .NET framework to be comprehensive in its capabilities to handle this but made as a general purpose utility for image processing orientated around - well, any purpose. Including brainiacs who utilize it to its full capacity (doing things so complex they make my nose bleed).

In regard to specific web use trivial implementations I found it to be confusing. Trivial things like scaling an image programatically (a seemingly simple task with all that power at my disposal.) When asking straightforward questions like "how do I scale the image down" - which is 11 web-relevance-o-meter - the learning curve is far too steep so web developers resort to ugly hacking like manually scaled images with file-naming conventions which relate scaled and full size images to one another (yuck).

That is not to say any web developer with sufficent time at their disposal would not be able to implement a dynamically sized solution themselves but the information is so dense via the official MSDN route and so scattered on random blogs and forums that when it comes to implementation quirks it took me almost a week just to: scale an image programatically, store it in a file with the correct codec. That was it. I started on a Monday morning. 333 blogs, forums and MSDN entries later I managed to finish it by Thursday afternoon.

In the aftermath I came up with a little helper which might save others time (although that is doubtful because they'll have to read the rest of this post). I found it to be useful in implementing actual requirements instead of dealing with Drawing namespace related problems which produce more Drawing namespace related problems. Also, by producing it, I found reading the code simultaneously dissects the inner workings of the Drawing's implementation details without overburdening the programmer with a flood of information. It is intended as a 'no frills, no fuss', simple implementation with focus only on scaling the image down and loading the image data so it can be used to instantiate other Drawing objects;  neat and tidy enough for someone unfamiliar with the details to go through without having to waste miles of time.

I'll also use this walk-through to try explain some lesser known (yet common?) Drawing namespace tools. Typically what I work with in web development are image files (produced by designers). In order to convert them to programatically accessible object instance references without fuss I had to jump through various hoops with some fancy footwork. The code that follows is the end result of a learning process that drained me and pushed my patience. Hopefully someone else can avoid that by using it.

What does it entail?

It ought to be as simple as

new Bitmap("wherever/the/file/is.jpg").Scale(doubleMaxSize);

When coding a top-down solution, my aim is to abstract the implementation details of dealing with the internal libraries primarily for the sake of legibility but also maintenance and re-use. From a web development perspective there needs to be a way to load the image binary stream from a file and return something workable i.e. an object instance which can be computationally manipulated in terms of dimension and compression parameters. Already here you can see the learning curve steeping upwards at an alarming rate.

What is 'workable' in Drawing space terms? It seems the data type du jour appears to be the abstract class Image. If you have already dabbled in some code attempting to dig your way out of the rabbit hole of image manipulation, you probably would have discovered this yourself with overloaded constructor for Bitmap class which takes Image instance as initial input (and, aptly, constructs a new instance of the Bitmap from it). I'll assume the file name and directory location is known (which it won't be IRL) and the reader is somewhat familiar with basics of System.IO. If not I think a search engine of your choice should suffice - these are straightforward and commonly used file operations that should be well documented. Plus this post is wordy.

So here we go:

FileStream tmp = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);

That much was easy enough. Pretty sure everyone who has stumbled on this entry has gotten this far. Now what? A way of converting the file stream into a byte array which represents an object of some kind.

The Image class has a static FromStream method which takes a stream as input and returns an Image reference. We can easily wrap this as an extension:

public static class Extensions { public static Image ToImage(this Stream value) { return Image.FromStream(value, true, true); } }

(Side note/pr0-tip: if this is your first time working with extension methods be sure to put the Extensions class in Extensions.cs in the root of your project or alternatively, the class extensions have to be in a namespace available to the compiler i.e. the extension has to be in the same namespace as the class where you want to use it, or you have to add a 'using' for it.)

Once this is working a ToImage method will be bound to the Stream class and, by extension (hehe - pun), to all its child classes; FileStream included. This allows us to deal with an instance of type Image:

FileStream tmp = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite); Image img = tmp.ToImage();

It is worthwhile mentioning that FileStream is a disposable class. It uses shared resources and it is not to be wielded wily-nilly or you might spend a few hours untangling a "bug" which is a line of code directly above that locked the file on the first run. Which means whatever we want to do with the file stream needs to be wrapped inside a using clause, or risk memory leaks and such nonsense (all of which distract from the problem at hand).

So really the code should look like:

using (FileStream tmp = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); }

And in the context of an actual handler:

using System.Drawing; using System.Drawing.Drawing2D; using System.IO; using System.Web; public class ImgHandler : IHttpHandler { SmoothingMode smoothing; InterpolationMode interpolation; PixelOffsetMode pixelOffset; public ImgHandler() { smoothing = SmoothingMode.HighSpeed; interpolation = InterpolationMode.HighQualityBilinear; pixelOffset = PixelOffsetMode.HighSpeed; } public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; byte[] imgArr = new byte[] { }; using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); } context.Response.BinaryWrite(imgArr); } }

We still have not addressed the core issue of scaling the image dimensions.

I used Rectangle object instance to draw the image into a new image object instance via Graphics class inversion control which, internally, performs the scaling routine:

Rectangle rect; using (Bitmap newImage = new Bitmap(rect.Width, rect.Height)) using (Graphics gr = Graphics.FromImage(newImage)) { gr.DrawImage(srcImage, rect); }

At this point we need to perform arithmetic which will produce a scaled version of newImage from srcImage into bounds set by Rectangle dimensions.

2D images are either "tall", "wide" or perfectly square which is to say the Rectangle will either have a width greater than its height, vice versa or they will be equal. This is grade school geometry arithmetic so let's not dwell on it.

public void Scale(Image srcImage, int max, System.Drawing.Drawing2DSmoothingMode smoothing, System.Drawing.Drawing2DInterpolationMode interpolation, System.Drawing.Drawing2DPixelOffsetMode pixelOffset) { Rectangle rect; if (srcImage.Height == srcImage.Width) rect = new Rectangle(0, 0, max, max); else if (srcImage.Height > srcImage.Width) rect = new Rectangle(0, 0, Convert.ToInt32(srcImage.Width * fraction(srcImage.Height, max)), max); else rect = new Rectangle(0, 0, max, Convert.ToInt32(srcImage.Height * fraction(srcImage.Width, max))); using (Bitmap newImage = new Bitmap(rect.Width, rect.Height)) using (Graphics gr = Graphics.FromImage(newImage)) { gr.SmoothingMode = smoothing; gr.InterpolationMode = interpolation; gr.PixelOffsetMode = pixelOffset; gr.CompositingQuality = CompositingQuality.HighQuality; gr.DrawImage(srcImage, rect); } } private static double fraction(int measure, double maxSize) { return (maxSize / measure); }

I have omitted an actual solution thus far to avoid confusing the issue of explaining the scaling process from passing image data around which - believe it or not - is actually the tricky bit.

It may seem easy but it is quite a hassle given all the stream closing and disposing. Which is why using clause is our best friend.

We can't return the instances of Graphic or the Bitmap because they are being disposed - and with good resource-related reason which I won't get into because I have no idea what is going on but found in practice to produce stack traces deep, deep down in the framework stack.

So we are left with performing some logic to get data out of the newImage instance in another way. To do this we'll need to get the data from newImage and we'll re-utilize the ToImage extension again:

public static byte[] ScaleDown(this Image srcImage, double maxSize) { return srcImage.ScaleDown(maxSize, System.Drawing.Drawing2D.SmoothingMode.AntiAlias, System.Drawing.Drawing2D.InterpolationMode.HighQualityBicubic, System.Drawing.Drawing2D.PixelOffsetMode.None); } public Image Scale(Image srcImage, int max, System.Drawing.Drawing2DSmoothingMode smoothing, System.Drawing.Drawing2DInterpolationMode interpolation, System.Drawing.Drawing2DPixelOffsetMode pixelOffset) { Rectangle rect; if (srcImage.Height == srcImage.Width) rect = new Rectangle(0, 0, max, max); else if (srcImage.Height > srcImage.Width) rect = new Rectangle(0, 0, Convert.ToInt32(srcImage.Width * fraction(srcImage.Height, max)), max); else rect = new Rectangle(0, 0, max, Convert.ToInt32(srcImage.Height * fraction(srcImage.Width, max))); using (Bitmap newImage = new Bitmap(rect.Width, rect.Height)) using (Graphics gr = Graphics.FromImage(newImage)) { gr.SmoothingMode = smoothing; gr.InterpolationMode = interpolation; gr.PixelOffsetMode = pixelOffset; gr.CompositingQuality = CompositingQuality.HighQuality; gr.DrawImage(srcImage, rect); using (var ms = new MemoryStream()) { newImage.Save(ms, srcImage.RawFormat); return ms.ToImage(); } } }

The new implementation should look like this:

public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; byte[] imgArr = new byte[] { }; using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); Image thumb = Scale(img, 100); } context.Response.BinaryWrite(imgArr); }

This brings us significantly closer. All that remains is to convert the thumb image to a byte array via a new memory stream and we are done!

public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; byte[] imgArr = new byte[] { }; using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); Image thumb = Scale(img, 100); using (var ms = new MemoryStream()) { thumb.Save(ms, img.RawFormat); imgArr = ms.GetBuffer(); } } context.Response.BinaryWrite(imgArr); }

And are we done yet!? Nope.

Those of you still reading are probably here because of GDI+ exception - Ye Olde Dreaded Scourge of The Seven Seas.

The problem (as far as I can tell) is that Image is an abstract class i.e. its intended use is only as a base class not as an object instance type. It might work up to a point because it is an easy way of referencing data types for different image compression/types on the stack but it ultimately fails when it comes trying to reference the actual data from the heap (which, I'm assuming, is what is happening on Save - somebody correct me because I did not bother looking it up).

So we might be tempted to revising into this:

using (var ms = new MemoryStream()) { thumb.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg); imgArr = ms.GetBuffer(); }

The alert reader has already noticed that the thumb image is derived from a Bitmap inside the Scale method. Hence the same error. How about:

using (var ms = new MemoryStream()) { thumb.Save(ms, System.Drawing.Imaging.ImageFormat.Bmp); imgArr = ms.GetBuffer(); }

BOOM! Again? Maybe MemoryBmp?

using (var ms = new MemoryStream()) { thumb.Save(ms, System.Drawing.Imaging.ImageFormat.MemoryBmp); imgArr = ms.GetBuffer(); }

'Value cannot be null. Parameter name: encoder.'

Oops. We don't really want to mess around with encoders at this point because it is leading us down the rabbit hole of solving internal framework problems which produce more internal framework problems to solve. What we want is what we set out to do - no frills, no fuss.

Where can we nip this in the bud? At the Scale level. We'll return raw binary data instead of an Image reference.

public byte[] Scale(Image srcImage, int max, System.Drawing.Drawing2DSmoothingMode smoothing, System.Drawing.Drawing2DInterpolationMode interpolation, System.Drawing.Drawing2DPixelOffsetMode pixelOffset) { Rectangle rect; if (srcImage.Height == srcImage.Width) rect = new Rectangle(0, 0, max, max); else if (srcImage.Height > srcImage.Width) rect = new Rectangle(0, 0, Convert.ToInt32(srcImage.Width * fraction(srcImage.Height, max)), max); else rect = new Rectangle(0, 0, max, Convert.ToInt32(srcImage.Height * fraction(srcImage.Width, max))); using (Bitmap newImage = new Bitmap(rect.Width, rect.Height)) using (Graphics gr = Graphics.FromImage(newImage)) { gr.SmoothingMode = smoothing; gr.InterpolationMode = interpolation; gr.PixelOffsetMode = pixelOffset; gr.CompositingQuality = CompositingQuality.HighQuality; gr.DrawImage(srcImage, rect); using (var ms = new MemoryStream()) { newImage.Save(ms, srcImage.RawFormat); return ms.GetBuffer(); } } }

Which solves the immediate problem. New implementation of ProcessRequest:

public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; byte[] imgArr = new byte[] { }; using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); imgArr = Scale(img, 100); } context.Response.BinaryWrite(imgArr); }

So why does the source image RawFormat work with Save when the image instance is loaded from a file stream but does not work when the image is an instance loaded from a memory stream?

Maybe the FromStream method sets the RawFormat correctly when loading from a FileStream because the file contains that information in the header.

Or perhaps the RawFormat cannot auto-magically infer the encoder when loaded from a MemoryStream thus resulting in the need for an actual encoder explicitly defined.

But probably not. Probably some other reason. Perhaps there is even a person working at Microsoft who knows this reason. I imagine this person to be wearing a very tall hat. I don't know why it doesn't work but the workaround works.

Yay! It's Miller time!

Welllllll... not quite.

Real World Practicalities

Suppose there is a failure in loading data from the file (e.g. the file is corrupt) or Scale craps out because you copypasta-ed it from some half-baked, shitty blog entry. What to do?

The bullet proof solution is to do this operation on Application_Start, loading the image data from a file you know exists (an image indicating a 404 would be a good request to put through to the design department) then store the byte array of that scaled image in a static read only field (located... somewhere - a static field of Global springs to mind) to be used as a default. If there is a problem with the image or the code it should be immediately obvious in testing phase as soon as the application is started.

That sorts the problem of default images but suppose you don't want to store the image byte array in memory. Suppose you want to scale the image once, store it on the file system, then have the handler check to see if the image exists in the file system to simply redirect browser request if it does.

The handler scales and stores the file only if the file doesn't exist. Reasons for this could be e.g. if the byte array is big because you are using Bitmap instead of a compressed format like jpeg. Or you're a pedant like I am.

Now you'd have to mess with codecs and such. You'd have to know image and file types and by the time you are done with all that, your image handler is violating the single responsibility principle so badly it makes my eyes bleed. So before you go down that road, refactor the code and break it up. We made it work and now is the perfect time to make it pretty (or at least prettier):

using System; using System.Drawing; using System.Drawing.Drawing2D; using System.IO; using System.Web; public class ImgHandler : IHttpHandler { public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; byte[] imgArr; using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); imgArr = img.Scale(100); } context.Response.BinaryWrite(imgArr); } public bool IsReusable { get { return false; } } } public static class Extensions { public static Image ToImage(this Stream value) { return Image.FromStream(value, true, true); } private static double fraction(int measure, double maxSize) { return (maxSize / measure); } public static byte[] Scale(this Image srcImage, int max) { return srcImage.Scale(max, SmoothingMode.HighSpeed, InterpolationMode.HighQualityBilinear, PixelOffsetMode.HighSpeed); } public static byte[] Scale(this Image srcImage, int max, SmoothingMode smoothing, InterpolationMode interpolation, PixelOffsetMode pixelOffset) { Rectangle rect; if (srcImage.Height == srcImage.Width) rect = new Rectangle(0, 0, max, max); else if (srcImage.Height > srcImage.Width) rect = new Rectangle(0, 0, Convert.ToInt32(srcImage.Width * fraction(srcImage.Height, max)), max); else rect = new Rectangle(0, 0, max, Convert.ToInt32(srcImage.Height * fraction(srcImage.Width, max))); using (Bitmap newImage = new Bitmap(rect.Width, rect.Height)) using (Graphics gr = Graphics.FromImage(newImage)) { gr.SmoothingMode = smoothing; gr.InterpolationMode = interpolation; gr.PixelOffsetMode = pixelOffset; gr.CompositingQuality = CompositingQuality.HighQuality; gr.DrawImage(srcImage, rect); using (var ms = new MemoryStream()) { newImage.Save(ms, srcImage.RawFormat); return ms.GetBuffer(); } } } }

There we go. A little abstraction, some overloading and now we can refactor and implement new features of the ProcessRequest as follows:

public class ImgHandler : IHttpHandler { readonly ImageCodecInfo _jpegCodecInfo = ImageCodecInfo .GetImageEncoders() .First(codec => codec.FormatID.Equals(ImageFormat.Jpeg.Guid)); public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; if (!File.Exists(context.Server.MapPath(@"~/sampleThumb.jpg"))) using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); using (MemoryStream ms = new MemoryStream(img.Scale(100))) using (Bitmap bmp = new Bitmap(ms.ToImage())) { EncoderParameters encoderParams = new EncoderParameters(1); encoderParams.Param[0] = new EncoderParameter(Encoder.Quality, 85L); bmp.Save(context.Server.MapPath(@"~/sampleThumb.jpg"), _jpegCodecInfo, encoderParams); } } if (File.Exists(context.Server.MapPath(@"~/sampleThumb.jpg"))) context.Response.Redirect("~/sampleThumb.jpg"); else context.Response.Redirect("~/_dne.jpg"); } public bool IsReusable { get { return false; } } }

Browsers cannot handle a 304 redirect from <img src/> attribute and so your handler will, in fact, have to stream the image binary data directly to the browser with an HTTP status code 200. However, loading it from a file is a much simpler excercise once the image has been scaled and the file generated:

public class ImgHandler : IHttpHandler { readonly ImageCodecInfo _jpegCodecInfo = ImageCodecInfo .GetImageEncoders() .First(codec => codec.FormatID.Equals(ImageFormat.Jpeg.Guid)); public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; if (!File.Exists(context.Server.MapPath(@"~/sampleThumb.jpg"))) using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); using (MemoryStream ms = new MemoryStream(img.Scale(100))) using (Bitmap bmp = new Bitmap(ms.ToImage())) { EncoderParameters encoderParams = new EncoderParameters(1); encoderParams.Param[0] = new EncoderParameter(Encoder.Quality, 85L); bmp.Save(context.Server.MapPath(@"~/sampleThumb.jpg"), _jpegCodecInfo, encoderParams); } } if (File.Exists(context.Server.MapPath(@"~/sampleThumb.jpg"))) using (var tmp = File.Open(context.Server.MapPath(@"~/sampleThumb.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite).ToImage()) using (var ms = new MemoryStream()) { tmp.Save(ms, tmp.RawFormat); context.Response.BinaryWrite(ms.GetBuffer()); } else using (var tmp = File.Open(context.Server.MapPath(@"~/_dne.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite).ToImage()) using (var ms = new MemoryStream()) { tmp.Save(ms, tmp.RawFormat); context.Response.BinaryWrite(ms.GetBuffer()); } } public bool IsReusable { get { return false; } } }

Refactoring Generic Solution

Now that is one ugly chunk code. Reading it is like reading Proust. By the time you get to the end of the clause, you have no idea what you started reading. Or why. Once again - extensions to the rescue.

Observant reader would have picked up on a pattern repeating itself in getting the byte array from a file stream. Let's abstract it into an extension of the Stream class.

public static byte[] ToImageBytes(this Stream input) { using (input) using (var tmp = input.ToImage()) using (var ms = new MemoryStream()) { tmp.Save(ms, tmp.RawFormat); return ms.GetBuffer(); } }

and while we're at it:

public static byte[] ToImageBytes(this Stream input) { using (input) return ToBytes(input.ToImage()); } private static byte[] ToBytes(this Image tmp) { return tmp.ToBytes(null); } private static byte[] ToBytes(this Image tmp, ImageFormat format) { using (tmp) using (var ms = new MemoryStream()) { tmp.Save(ms, format ?? tmp.RawFormat); return ms.GetBuffer(); } }

So Scale can now become:

Rectangle rect; if (srcImage.Height == srcImage.Width) rect = new Rectangle(0, 0, max, max); else if (srcImage.Height > srcImage.Width) rect = new Rectangle(0, 0, Convert.ToInt32(srcImage.Width * fraction(srcImage.Height, max)), max); else rect = new Rectangle(0, 0, max, Convert.ToInt32(srcImage.Height * fraction(srcImage.Width, max))); using (Bitmap newImage = new Bitmap(rect.Width, rect.Height)) using (Graphics gr = Graphics.FromImage(newImage)) { gr.SmoothingMode = smoothing; gr.InterpolationMode = interpolation; gr.PixelOffsetMode = pixelOffset; gr.CompositingQuality = CompositingQuality.HighQuality; gr.DrawImage(srcImage, rect); return newImage.ToBytes(srcImage.RawFormat); }

And, finally at long last, ProcessRequest can become:

public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; if (!File.Exists(context.Server.MapPath(@"~/sampleThumb.jpg"))) using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); using (MemoryStream ms = new MemoryStream(img.Scale(100))) using (Bitmap bmp = new Bitmap(ms.ToImage())) { EncoderParameters encoderParams = new EncoderParameters(1); encoderParams.Param[0] = new EncoderParameter(Encoder.Quality, 85L); bmp.Save(context.Server.MapPath(@"~/sampleThumb.jpg"), _jpegCodecInfo, encoderParams); } } File buffer; if (File.Exists(context.Server.MapPath(@"~/sampleThumb.jpg"))) buffer = File.Open(context.Server.MapPath(@"~/sampleThumb.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite); else buffer = File.Open(context.Server.MapPath(@"~/_dne.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite); context.Response.BinaryWrite(buffer.ToImageBytes()); }

Putting It All Together

This implementation results in readable code for tackling the problem at hand, as well as robust and reusable methods which can be made part of a utility library which, in turn, can be included in other projects with similar requirements. What is important and of immediate concern is loading of image data, scaling image dimensions and serving the binary data is done concisely with all the relevant references exposed in readable portions.

using System; using System.Drawing; using System.Drawing.Drawing2D; using System.Drawing.Imaging; using System.IO; using System.Linq; using System.Web; public class ImgHandler : IHttpHandler { readonly ImageCodecInfo _jpegCodecInfo = ImageCodecInfo .GetImageEncoders() .First(codec => codec.FormatID.Equals(ImageFormat.Jpeg.Guid)); public void ProcessRequest(HttpContext context) { context.Response.ContentType = "image/jpg"; if (!File.Exists(context.Server.MapPath(@"~/sampleThumb.jpg"))) using (FileStream tmp = File.Open(context.Server.MapPath(@"~/sample.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { Image img = tmp.ToImage(); using (MemoryStream ms = new MemoryStream(img.Scale(100))) using (Bitmap bmp = new Bitmap(ms.ToImage())) { EncoderParameters encoderParams = new EncoderParameters(1); encoderParams.Param[0] = new EncoderParameter(Encoder.Quality, 85L); bmp.Save(context.Server.MapPath(@"~/sampleThumb.jpg"), _jpegCodecInfo, encoderParams); } } byte[] buffer; if (File.Exists(context.Server.MapPath(@"~/sampleThumb.jpg"))) buffer = File.Open(context.Server.MapPath(@"~/sampleThumb.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite) .ToImageBytes(); else buffer = File.Open(context.Server.MapPath(@"~/_dne.jpg"), FileMode.Open, FileAccess.Read, FileShare.ReadWrite) .ToImageBytes(); context.Response.BinaryWrite(buffer); } public bool IsReusable { get { return false; } } } public static class Extensions { public static Image ToImage(this Stream value) { return Image.FromStream(value, true, true); } public static byte[] ToImageBytes(this Stream input) { using (input) return ToBytes(input.ToImage()); } private static byte[] ToBytes(this Image tmp) { return tmp.ToBytes(null); } private static byte[] ToBytes(this Image tmp, ImageFormat format) { using (tmp) using (var ms = new MemoryStream()) { tmp.Save(ms, format ?? tmp.RawFormat); return ms.GetBuffer(); } } private static double fraction(int measure, double maxSize) { return (maxSize / measure); } public static byte[] Scale(this Image srcImage, int max) { return srcImage.Scale(max, SmoothingMode.HighSpeed, InterpolationMode.HighQualityBilinear, PixelOffsetMode.HighSpeed); } public static byte[] Scale(this Image srcImage, int max, SmoothingMode smoothing, InterpolationMode interpolation, PixelOffsetMode pixelOffset) { Rectangle rect; if (srcImage.Height == srcImage.Width) rect = new Rectangle(0, 0, max, max); else if (srcImage.Height > srcImage.Width) rect = new Rectangle(0, 0, Convert.ToInt32(srcImage.Width * fraction(srcImage.Height, max)), max); else rect = new Rectangle(0, 0, max, Convert.ToInt32(srcImage.Height * fraction(srcImage.Width, max))); using (Bitmap newImage = new Bitmap(rect.Width, rect.Height)) using (Graphics gr = Graphics.FromImage(newImage)) { gr.SmoothingMode = smoothing; gr.InterpolationMode = interpolation; gr.PixelOffsetMode = pixelOffset; gr.CompositingQuality = CompositingQuality.HighQuality; gr.DrawImage(srcImage, rect); return newImage.ToBytes(srcImage.RawFormat); } } }

When you see it like that it looks simple but when you look at it from the seemingly neverending library of methods, properties and enums which are part of the Drawing space... it doesn't?

(I stated that because... I can?)

A variety of image types would require a more complex solution; one where a collection of codecs needs to be individually implemented along with corresponding ContentType values.

[csharpandsuch]

Nullable Dictionary

DictionaryNullable is a (niche appeal) class written to wrap the logic of checking for existence of keys to a conditional to check if the value is null, for circumstances where

the values are never null (or the value type doesn't permit nulls) or

the presence of a key or its absence is logically indistinguishable and irrelevant from the value being null e.g.

var myDict = new DictionaryNullable<string, string> {{"key1", "value1"}}; string val = (myDict["key2"] ?? (myDict["key2"] = "value2"));

It extends Dictionary and overrides the collection reference operator []:

public sealed class DictionaryNullable<TK, T> : Dictionary<TK, T> { public new T this[TK key] { get { if (ContainsKey(key)) return base[key]; return default(T); } set { base[key] = value; } } }

Complete class code:

public sealed class DictionaryNullable<TK, T> : Dictionary<TK, T> where TK : IComparable<TK> { public static DictionaryNullable<TK, T> FromDictionary(Dictionary<TK, T> actual) { return new DictionaryNullable<TK, T>(actual); } public Dictionary<TK, T> ToDictionary() { return new Dictionary<TK, T>(this); } public DictionaryNullable() : base() { } public DictionaryNullable(SerializationInfo info, StreamingContext context) : base(info, context) { } public DictionaryNullable(int capacity) : base(capacity) { } public DictionaryNullable(int capacity, IEqualityComparer<TK> iEq) : base(capacity, iEq) { } public DictionaryNullable(IDictionary<TK, T> iDic, IEqualityComparer<TK> iEq) : base(iDic, iEq) { } public DictionaryNullable(IDictionary<TK, T> iDic) : base(iDic) { } public new T this[TK key] { get { if (ContainsKey(key)) return base[key]; return default(T); } set { base[key] = value; } } }

[csharpandsuch]

Serializing Dictionaries

If you found the serialization post useful, you are probably leveraging XML and serialization so you might also find this simple extension for serializable dictionaries useful.

using System.Collections.Generic; using System.Xml; using System.Xml.Schema; using System.Xml.Serialization; [XmlRoot("dictionary")] public class SerializableDictionary<TKey, TValue> : Dictionary<TKey, TValue>, IXmlSerializable { public SerializableDictionary() : base() { } public SerializableDictionary(IDictionary<TKey, TValue> enumerable) : base(enumerable) { } #region IXmlSerializable Members public XmlSchema GetSchema() { return null; } public void ReadXml(XmlReader reader) { XmlSerializer keySerializer = new XmlSerializer(typeof(TKey)); XmlSerializer valueSerializer = new XmlSerializer(typeof(TValue)); bool wasEmpty = reader.IsEmptyElement; reader.Read(); if (wasEmpty) return; while (reader.NodeType != XmlNodeType.EndElement) { reader.ReadStartElement("item"); reader.ReadStartElement("key"); TKey key = (TKey)keySerializer.Deserialize(reader); reader.ReadEndElement(); reader.ReadStartElement("value"); TValue value = (TValue)valueSerializer.Deserialize(reader); reader.ReadEndElement(); Add(key, value); reader.ReadEndElement(); reader.MoveToContent(); } reader.ReadEndElement(); } public void WriteXml(XmlWriter writer) { XmlSerializer keySerializer = new XmlSerializer(typeof(TKey)); XmlSerializer valueSerializer = new XmlSerializer(typeof(TValue)); foreach (TKey key in Keys) { writer.WriteStartElement("item"); writer.WriteStartElement("key"); keySerializer.Serialize(writer, key); writer.WriteEndElement(); writer.WriteStartElement("value"); TValue value = this[key]; valueSerializer.Serialize(writer, value); writer.WriteEndElement(); writer.WriteEndElement(); } } #endregion }

[csharpandsuch]

XML Serialize/Deserialize

Simplification of XML serializing/deserializing for fun and profit is (relatively) easy and not-particularly-but-somewhat useful in certain circumstances. It need not be complex as it is well documented and relatively straightforward to figure out yourself but I like code re-use so I wrapped the common case where we are not concerned with XML namespaces and other complexities. So here is a wrapper abstraction for the XmlSerializer class which helps serializing/deserializing  using the generic tuple specifier in the Serializer class:

using System; using System.Collections.Generic; using System.Xml.Serialization; using System.IO; using System.Xml; public abstract class SerializerBase { protected static readonly Dictionary<Type, XmlSerializer> Serializers = new Dictionary<Type, XmlSerializer>(); } public class Serializer<T> : SerializerBase { //XmlSerializer works by generating an assembly on the fly; //if you use the XmlSerializer(Type) constructor, it caches it //and looks it up; but for any other constructor it doesn't. //And assemblies can't (usually) be unloaded. So you just get //more and more and more assemblies eating your memory. //need to cache the serializer if you are running this in a loop //private static readonly Dictionary<Type, XmlSerializer> serializers = new Dictionary<Type, XmlSerializer>(); public static string Serialize(T obj) { return Serialize(obj, new Type[] { }); } public static string Serialize(T obj, Type[] extraTypes) { return _serialize(obj, _makeSerializer(typeof(T), extraTypes)); } public static T Deserialize(string xml) { return Deserialize(xml, new Type[] { }); } public static T Deserialize(string xml, Type[] extraTypes) { if (string.IsNullOrEmpty(xml)) throw new ArgumentNullException("xml"); XmlSerializer serializer = _makeSerializer(typeof (T), extraTypes); using (StringReader stream = new StringReader(xml)) { try { return (T)serializer.Deserialize(stream); } catch (Exception ex) { throw new InvalidOperationException("Failed to create object from XML string.", ex); } } } private static XmlSerializer _makeSerializer(Type T, Type[] extraTypes) { if (extraTypes.Length <= 0) return new XmlSerializer(T); XmlSerializer xmlSerializer; if (Serializers.TryGetValue(T, out xmlSerializer)) return xmlSerializer; Serializers.Add(T, new XmlSerializer(T, extraTypes)); return Serializers[T]; } private static String _serialize(T obj, XmlSerializer serializer) { StringWriter stringWriter = new StringWriter(); try { using (XmlTextWriter xmlWriter = new XmlTextWriter(stringWriter) { Formatting = Formatting.None }) { xmlWriter.WriteRaw(""); XmlSerializerNamespaces xmlNameSpace = new XmlSerializerNamespaces(); xmlNameSpace.Add("", null); serializer.Serialize(xmlWriter, obj, xmlNameSpace); stringWriter.Flush(); return stringWriter.ToString(); } } catch (Exception ex) { throw new InvalidOperationException("Failed to create XML string from object.", ex); } finally { stringWriter.Close(); stringWriter.Dispose(); } } }

What Does Implementation Look Like

Suppose we have a trivial class of objects Customer.

using System.Xml.Serialization; [XmlRoot(ElementName = "customer")] public class Customer { [XmlAttribute(AttributeName = "id")] public int ID; [XmlElement] public string Name; [XmlElement] public string Surname; }

Then the implementation of the Serializer<T> for Customer would look like this:

Customer customer = Serializer<Customer>.Deserialize("<customer id=\"123\"><Name>Joe</Name><Surname>Soap</Surname></customer>"); Console.WriteLine(customer.ID); //123

We can create an instance of an object of type customer from a hard-coded string as above an reverse the process by using serialize as follows:

string strCustomerXml = Serializer<Customer>.Serialize(customer); Console.WriteLine(strCustomerXml); //<customer id="123"><Name>Joe</Name><Surname>Soap</Surname></customer>

So far this is all pretty straightforward and simple in the trivial cases where serialization is not really all that needed. Which brings up the question -

How Is This Useful?

The short answer is "it isn't"; as the old berates about XML go "if you have a problem you might be tempted to solve it with XML - now you have 2 problems" and "XML is like violence - if it's not working then you are not using enough".

I forget who these quotes are attributed to but it matters little as with all knee-jerks it is both truth and deception depending on the circumstances we are dealing with and how we deal with them.

In my experience, XML is Good EnoughTM way of representing hierarchical data structures in systems where some human-readable abstraction is helpful for verification. In web CMS systems, for example, it's a good way of storing and translating discrete data which can later be converted to and from HTML, which is (supposed to be) a subset of XML.

Let's expand on the Customer class and make it slightly more complex so we can see why the Deserialize/Serialize methods are overloaded. Suppose the Customer class contains a property of a type which is another custom class as such:

using System.Collections.Generic; using System.Xml.Serialization; [XmlRoot(ElementName = "customer")] public class Customer { [XmlAttribute(AttributeName = "id")] public int ID; [XmlElement] public string Name; [XmlElement] public string Surname; [XmlElement(ElementName = "invoice")] public List<Invoice> Invoices = new List<Invoice>(); } [XmlRoot(ElementName = "invoice")] public class Invoice { [XmlAttribute(AttributeName = "id")] public int ID; [XmlArray(ElementName = "products")] [XmlArrayItem(ElementName = "product")] public List<Product> Items = new List<Product>(); } [XmlRoot(ElementName = "product")] public class Product { [XmlAttribute(AttributeName = "id")] public int ID; [XmlElement] public string Name; [XmlElement] public string Description; }

We can see here that the hierarchy has expanded and there are multiple hierarchies represented in a single class type Customer.

So how is that helpful?

We are representing complex relations which in RDBMS SQL structure would be represented by normalized tables Customer, Product and Invoice.

Then a linking table InvoiceProduct containing foreign keys to the Invoice and Product table as well as a foreign key in the Invoice table to the Customer primary key, in order to represent the [0,*] relationship between customers and invoices (in this case Product doesn't represent stock but rather the product on offer).

This is a complex relational structure which would need to be resolved programatically in some way.

If we are using OO, hierarchical data as above is one way of resolving it. Another would be object-relational mapping (storing data in RDBMS and manually populating object properties from SQL table fields or using an automated ORM system e.g. NHibernate, Entity Framework).

Where this approach is helpful (even with object-relational mapping storage) is that when presenting the data as XML via a web service, some form of conversion from object data types to XML would need to take place. Using this approach simplifies the final step of outputting the data in the XML format.

So a meaningful use would be if you have a particular set of requirements where the object class definitions serve as a programmatic intermediary for computation but also for XML output.

Unrelated side note: I also store settings with a utility class which implements this to serialize objects from an XML file in the ~/App_Data folder; which allows me to play with non-environment, cross-server agnostic but app specific settings between deployment servers (for testing, staging and production). I don't have to worry about overwriting environment specific settings in web.config e.g. different connection strings and location/@path security while the XML is in a human-readable and, for an HTML techy, familiar markup format (editable on the production server, if need be, with any text editor).

In Action

OK so lets actually deal with the problem of deserializing the Customer class. The XML now might look like this:

<root> <customer id="123"> <Name>Joe</Name> <Surname>Soap</Surname> <invoice id="456"> <products/> </invoice> <invoice id="789"> <products> <product id="456"> <Name>SoapX</Name> <Description>Soap X: The soap-on-a-rope that makes soaping "dope"!</Description> </product> </products> </invoice> </customer> <customer id="90210"> <Name>Jane</Name> <Surname>Doe</Surname> </customer> </root>

2 customers. Soap, Joe (id 123) and Doe, Jane (id 90210) where Jane has no invoices and Joe has 2 invoices, 456 and 789.

One containing an empty list of products and the other with a list of products containing one item: product SoapX (id 123).

In order to convert to and from the XML representation of Customer, the Serializer<Customer> utility needs to be passed the complex types of the Customer properties (namely Invoice) as well as the complex types of the properties of the sub-properties (so for Invoice property, Product). The rabbit hole stops at some sub-sub-... sub-property. For this example at Product but if Product contained other complex types (e.g. Stock), those would need to be included in the array.

Before we see this illustrated with code, let's store the XML in a separate file Example.xml and load the instance of the objects from the file.

System.XmlXmlDocument xmlDoc = new System.XmlXmlDocument(); xmlDoc.Load("[full-path-to]/Example.xml"); var nodes = xmlDoc.DocumentElement.SelectNodes("customer"); for (int i = 0; i < nodes.Count; i++) { var node = nodes.Item(i); Customer customer = Serializer<Customer>.Deserialize(node.OuterXml, new Type[] { typeof(Invoice), typeof(Product) }); Console.WriteLine(customer.ID); string strCustomerXml = Serializer<Customer>.Serialize(customer, new Type[] { typeof(Invoice), typeof(Product) }); Console.WriteLine(strCustomerXml); }

This produces:

123 <customer id="123"><Name>Joe</Name><Surname>Soap</Surname><invoice id="456"><products /></invoice><invoice id="789"><products><product id="456"><Name>SoapX</Name><Description>Soap X: The soap-on-a-rope that makes soaping "dope"!</Description></product></products></invoice></customer> 90210 <customer id="90210"><Name>Jane</Name><Surname>Doe</Surname></customer>

So it is fairly simple and straight forward.

Also, it's rather cumbersome keeping track of all the additional types per method call. It is tedious and, more alarmingly, error prone.

Going forward, we might introduce additional types and then we'd have to update all the implementations. We would not get a compile time exception if we missed any and, worse, we wouldn't even get any runtime exceptions either as the data would simply be omitted from being processed on either serializing or deserializing to and from XML.

For these reasons I introduce another abstraction level to automate additional type properties as soon as the modifications to any of the classes in the hierarchy chain occur. This way, adding a new property of a new type to any classes in the chain will automatically propagate to all the input and output serializing points.

using System; using System.Linq; using System.Xml.Serialization; [Serializable] public abstract class SimpleSerializable<T> where T : SimpleSerializable<T>, new() { private readonly T _raw = default(T); // Static Fields In a Generic Type // it's ok to have a static field in a generic type // so long as we get one field per type argument. // In the vast majority of cases, having a static field // in a generic type is a sign of an error. // The reason for this is that a static field in a generic type // will not be shared among instances of different close constructed types. // This means that for a generic class SimpleSerializable<T>, values of // SimpleSerializable<Customer>.types and SimpleSerializable<Customer>.types // have different, independent values. In this case, this is not only preferable // but actually unavoidable. private static readonly Type[] types = new Type[] { }; static SimpleSerializable() { types = typeof(T).Subtypes(new Type[] { }).ToArray(); } protected SimpleSerializable() { if (_raw == null) _raw = (T)Convert.ChangeType(this, typeof(T)); } protected SimpleSerializable(string representation) { _raw = Serializer<T>.Deserialize(representation, types); } public T Object() { return _raw; } public override string ToString() { return Serializer<T>.Serialize(_raw, types); } }

The key benefit of this abstraction, is the Subtypes extension which loads all the relevant property complex types on startup when the application is initially loaded.

It's worthwhile disclosing that I'm out of my depth when it comes to reflection bindings and suchlike core .NET/IL/CLR... er... stuff so take this extension with a pinch of salt and modify it as you encounter bugs (especially the excluded variable) which is how I got it working to being with.

It works but has not been field tested in any other field except my own UoD of web dev and web services. Nonetheless, here's a single-field-tested version:

public static class TypeExtensions { public static IEnumerable<Type> Subtypes(this Type typ, Type[] state) { Type[] excluded = new Type[] { typeof(Array), typeof(Type[]), typeof(Object), typeof(string), typeof(XmlNodeList) }; Type[] initial = Subtypes(typ) .Where(itm => !itm.IsPrimitive & (itm.TypeInitializer == null || itm.TypeInitializer.IsConstructor) & !itm.IsCOMObject & !itm.IsArray & itm.IsClass & !itm.IsEnum & !excluded.Contains(itm)) .ToArray(); foreach (Type type in initial) if (!state.Contains(type) & type.Subtypes().Any(t => t.IsAssignableFrom(typeof(ISerializable)))) { initial = initial .Concat(Subtypes(type, initial.Concat(state).Distinct().ToArray())) .Distinct() .ToArray(); } return state.Concat(initial); } private static IEnumerable<Type> Subtypes(this Type typ) { return typ .GetProperties(System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Instance) .MergeWith(typ.GetFields(System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Instance), prp => prp.PropertyType, fld => fld.FieldType); } }

Putting It All Together

So now we are free to define the Customer class as an extension of the SimpleSerializable, which will allow us to utilize it's initializer and ToString to set with and get XML format representation of the object instance without worrying about omitting complex subtypes in the properties. This means modifying the Customer class as follows:

[XmlRoot(ElementName = "customer")] public class Customer : SimpleSerializable<Customer> { public Customer() { } public Customer(string strObjXml) : base(strObjXml) { } [XmlAttribute(AttributeName = "id")] public int ID; [XmlElement] public string Name; [XmlElement] public string Surname; [XmlElement(ElementName = "invoice")] public List<Invoice> Invoices = new List<Invoice>(); }

The implementation can now be:

System.XmlXmlDocument xmlDoc = new System.XmlXmlDocument(); xmlDoc.Load("[full-path-to]/Example.xml"); var nodes = xmlDoc.DocumentElement.SelectNodes("customer"); for (int i = 0; i < nodes.Count; i++) { var node = nodes.Item(i); Customer customer = new Customer(node.OuterXml).Object(); Console.WriteLine(customer.ID); string strCustomerXml = customer.ToString(); Console.WriteLine(strCustomerXml); }

[csharpandsuch]

Web Response Wrapper

The System.Net namespace is a helpful library containing tools to network. In this post we'll wrap HttpWebResponse and WebResponse streaming of data for HTTP requests in a utility class HttpRemoteCall.

Nuts-n-Bolts:

public enum HttpMethods { GET = 0, POST = 1, DELETE = 2, PUT = 3 } public class HttpRemoteCall { public int TimeoutMilisecs = 5000; public HttpMethods HttpMethod = HttpMethods.GET; public string NetworkUserID; public string NetworkPassword; private HttpWebRequest _request; private readonly string _url; public HttpRemoteCall(string url) { _url = url; } public HttpRemoteCall(string url, string networkUser, string networkPin) : this(url) { NetworkUserID = networkUser; NetworkPassword = networkPin; } }

The basics of the class are the default values and network credentials required to make a web request.

Some logic to construct the WebRequest instance:

public void Create(WebHeaderCollection headers, string send) { _request = (HttpWebRequest)WebRequest.Create(_url); _request.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.BypassCache); bool hasCredentials = !string.IsNullOrEmpty(NetworkUserID); if (hasCredentials) _request.Credentials = new NetworkCredential(NetworkUserID, NetworkPassword); if (!string.IsNullOrEmpty(send) && HttpMethod == HttpMethods.GET) { //default to POST HttpMethod = HttpMethods.POST; } _request.Method = HttpMethod.ToString(); _request.ReadWriteTimeout = TimeoutMilisecs; _request.Timeout = TimeoutMilisecs; _request.Headers = headers; } public void Create() { Create(new WebHeaderCollection(), null); } public void Create(string send) { Create(new WebHeaderCollection(), send); } public void Create(WebHeaderCollection headers) { Create(headers, null); }

So far, so good. This is what a trivial implementation might look like.

HttpRemoteCall remoteWS = new HttpRemoteCall("http://google.com"); remoteWS.Create();

How To Post To A Remote Server

We need to amend the Create method to handle the writing of data. To do this we will create a postToRemote method where we will assume UTF8 encoding which is a starting point for extensions into multiple encodings which I have not found a use for myself so have not implemented.

So the core functionality will look like:

private void postToRemote(string send) { byte[] b = new byte[] {}; if (!string.IsNullOrEmpty(send)) { b = Encoding.UTF8.GetBytes(send); } _request.ContentLength = b.Length; _request.GetRequestStream().Write(b, 0, b.Length); _request.GetRequestStream().Flush(); _request.GetRequestStream().Close(); } public void Create(WebHeaderCollection headers, string send) { _request = (HttpWebRequest)WebRequest.Create(_url); bool hasCredentials = !string.IsNullOrEmpty(NetworkUserID); if (hasCredentials) _request.Credentials = new NetworkCredential(NetworkUserID, NetworkPassword); if (!string.IsNullOrEmpty(send) && HttpMethod == HttpMethods.GET) { //default to POST HttpMethod = HttpMethods.POST; } _request.Method = HttpMethod.ToString(); _request.ReadWriteTimeout = TimeoutMilisecs; _request.Timeout = TimeoutMilisecs; _request.Headers = headers; if (HttpMethod != HttpMethods.GET) { postToRemote(send); } }

This is all cool but it lacks some basic stuff like cache policy as well as content type and accept headers (as well as other header options but those are the most common ones I found re-occur in patterns for web remote calls).

So we need to define:

public class HttpRemoteCall { public enum WebRemoteCallAccepts { //[Description("text/xml")] Default = 1, //[Description("text/html, application/xhtml+xml, */*")] StdHttp = 2 } public WebRemoteCallAccepts Accepts = WebRemoteCallAccepts.StdHttp; public enum WebRemoteCallContentTypes { //[Description("text/xml;charset=\"utf-8\"")] Default = 1, //[Description("text/html;charset=\"utf-8\"")] StdHttp = 2 } public WebRemoteCallContentTypes ContentType = WebRemoteCallContentTypes.StdHttp; public int TimeoutMilisecs = 5000; public HttpMethods HttpMethod = HttpMethods.GET; public string NetworkUserID; public string NetworkPassword; private HttpWebRequest _request; private readonly string _url; public HttpRemoteCall(string url) { _url = url; } public HttpRemoteCall(string url, string networkUser, string networkPin) : this(url) { NetworkUserID = networkUser; NetworkPassword = networkPin; } public void Create() { Create(new WebHeaderCollection(), null); } public void Create(string send) { Create(new WebHeaderCollection(), send); } public void Create(WebHeaderCollection headers) { Create(headers, null); } public void Create(WebHeaderCollection headers, string send) { _request = (HttpWebRequest)WebRequest.Create(_url); _request.CachePolicy = new System.Net.CacheHttpRequestCachePolicy(System.Net.Cache.HttpRequestCacheLevel.BypassCache); bool hasCredentials = !string.IsNullOrEmpty(NetworkUserID); if (hasCredentials) _request.Credentials = new NetworkCredential(NetworkUserID, NetworkPassword); if (!string.IsNullOrEmpty(send) && HttpMethod == HttpMethods.GET) { //default to POST HttpMethod = HttpMethods.POST; } _request.Method = HttpMethod.ToString(); _request.ReadWriteTimeout = TimeoutMilisecs; _request.Timeout = TimeoutMilisecs; _request.Headers = headers; _request.ContentType = ContentType.Description(); _request.Accept = Accepts.Description(); if (HttpMethod != HttpMethods.GET) { postToRemote(send); } } //etc... } //... public static class Extensions { public static string Description(this Enum value) { string name = Enum.GetName(value.GetType(), value); System.ReflectionFieldInfo field = value.GetType().GetField(name); if (field != null) { DescriptionAttribute attr = Attribute.GetCustomAttribute(field, typeof(DescriptionAttribute)) as DescriptionAttribute; return attr != null ? attr.Description : name.AddSpaces(); } return name.AddSpaces(); } }

And we'll need to do some error handling into the posting method:

//... //part of HttpRemoteCall private void postToRemote(string send) { byte[] b = new byte[] {}; if (!string.IsNullOrEmpty(send)) { b = Encoding.UTF8.GetBytes(send); } _request.ContentLength = b.Length; try { _request.GetRequestStream().Write(b, 0, b.Length); _request.GetRequestStream().Flush(); _request.GetRequestStream().Close(); } catch (Exception e) { if (Handy.TypeTester(e.GetType(), new Type[] { typeof (ArgumentException), typeof (ArgumentNullException), typeof (ArgumentOutOfRangeException), typeof (IOException) })) { throw new NetworkException(string.Format("Request stream write fail =>{0}", e.Message), e.InnerException); } if (Handy.TypeTester(e.GetType(), new Type[] { typeof (ProtocolViolationException), typeof (InvalidOperationException), typeof (WebException) })) { throw new NetworkException(string.Format("Request send fail =>{0}", e.Message), e.InnerException); } throw; } } //... public class NetworkException : ApplicationException { public NetworkException(string message) : base(message) { } public NetworkException(string message, Exception inner) : base(message, inner) { } } //... public static class Handy { public static bool TypeTester(Type isType, IEnumerable<Type> testTypes) { return testTypes.Any(testType => testType == isType); } }

Such that the implementation will be:

HttpRemoteCall remoteWS = new HttpRemoteCall("http://google.com"); try { remoteWS.Create(); } catch (NetworkException e) { Console.WriteLine(e); }

Putting It All Together

That takes care of the request construct. Now we need to retrieve and parse the response.

We'll wrap the response data in a custom class:

public class HttpRemoteResponse { public string Raw; public int StatusCode; public string ContentType; public List<KeyValuePair<string, string>> Headers; public string CharSet; }

Then add a method to HttpRemoteCall to stream the content from the remote server and output an object of this type:

public HttpRemoteResponse Response() { HttpRemoteResponse returnResponse = new HttpRemoteResponse { Raw = null }; WebResponse webResponse = null; try { webResponse = _request.GetResponse(); } catch (WebException wex) { if (webResponse != null) webResponse.Close(); HttpWebResponse httpResponse = wex.Response as HttpWebResponse; if (httpResponse != null) returnResponse.StatusCode = (int)httpResponse.StatusCode; //infer default Internal Server fault returnResponse.StatusCode = (int)HttpStatusCode.InternalServerError; return returnResponse; } //infer default bad request returnResponse.StatusCode = (int)HttpStatusCode.BadRequest; using (HttpWebResponse response = (HttpWebResponse)webResponse) { //get actual returnResponse.StatusCode = (int)response.StatusCode; using (Stream stream = response.GetResponseStream()) { try { if (stream == null) { //infer... returnResponse.StatusCode = (int)HttpStatusCode.NoContent; } //don't handle catch; auto re-throw if error returnResponse.Raw = new StreamReader(stream).ReadToEnd(); returnResponse.ContentType = response.ContentType; returnResponse.Headers = webResponseHeaders .AllKeys .Select(headerKey => new KeyValuePair<string, string>(headerKey, webResponse.Headers[(string)headerKey])) .ToList(); returnResponse.CharSet = response.CharacterSet; } finally { webResponse.Close(); } } } return returnResponse; }

Such that the implementation becomes:

HttpRemoteCall remoteWS = new HttpRemoteCall("http://google.com"); try { remoteWS.Create(); } catch (NetworkException e) { Console.WriteLine(e); } response = remoteWS.Response(); switch ((HttpStatusCode)response.StatusCode) { case HttpStatusCode.OK: Console.WriteLine("Hunky-Dorry"); break; case HttpStatusCode.InternalServerError: case HttpStatusCode.BadRequest: case HttpStatusCode.NoContent: Console.WriteLine(response.StatusCode); Console.WriteLine("internally inferred status codes - investigate response object further -> is server returning status code or is remote call inferring status code..."); break; default: Console.WriteLine(response.StatusCode); //TODO: handle each individually? break; } Console.WriteLine(response.Raw);