Thursday, August 27, 2015

Getting Lazy with LINQ, Json.Net and WebAPI

I've been working on a project that uses a WebAPI hosted via OWIN inside a Win32 client application. Yes, inside a client application. Forgive me. Let's not quibble about it.

My Controller Actions generally return an IEnumerable<T>, with T varying depending on the Action. Most of my Actions return the (raw) result of a LINQ query, which is convertible to IEnumerable<T>. A contrived example;

[HttpGet]
public IEnumerable<ApiCustomer> GetAllCustomers()
{
return (from customer
in _CustomerRepository.GetAll
where c.IsPublic
select GetApiCustomerFromFullCustomer(customer));
}




All was well until I needed to return an IEnumerable<BaseType> with the elements as different derived types. To enable this, you must tell the serialisation formatter to include type information. This allows the client to determine what type to create and populate from the serialised data. Luckily, the awesome Json.Net solves this problem. Just configure your default serialiser to include the type information;

config.Formatters.JsonFormatter.SerializerSettings.TypeNameHandling = TypeNameHandling.Auto;



TypeNameHandling.Object should also work. The difference is Auto is only adds the type information where it detects a difference in the types used. Object always includes the type information.

Unfortunately, this is where I came unstuck. I have declared a return type of IEnumerable<T>, but I'm not returning one. Sort of. IEnumerable<T> is an interface, so you can't actually return an instance of it. You can, and I do, return something that is castable/convertible to it.

The result of a LINQ statement is actually a generic type, created by the compiler at runtime. If you check the type at runtime you'll see a name like "WhereSelectIterator`2". This object instance is an enumerator (IEnumerator<T> or IEnumerator) that allows lazy enumeration of the query. It is also convertible to IEnumerable<T>, so without type information (de)serialises fine.

Json.Net serialises the LINQ result with type information of "WhereSelectIterator`2" (or similar). Presumably because "WhereSelectIterator`2" isn’t exactly IEnumerable<T>. When the client deserialises the data it can't find "WhereSelectIterator`2". That type only existed at runtime on the server, it's not defined in any assembly on the client. This results in an exception being thrown.

Json.Net already knows how to serialise some types when returned as IEnumerable<T>. Lists and arrays work (serialised as a Json array), and maybe other collection types too. Because of that behaviour, I had expected anything convertible to IEnumerable<T> would do the same. Unfortunately the enumerators from LINQ queries aren't treated the same way. I'm not sure if this is a bug in Json.Net. I haven't checked the Json.Net code, so I don't know what it actually does in detail. I'm also not sure if there are use cases I'm unaware of that might depend on this behaviour. It’s possible this behaviour is by design. Either way, I had a problem.

I needed a solution. The obvious one, often given on the interwebs, is to "just add ToList or ToArray" to the LINQ queries, so they return a type that exists on the client. This does work, but it didn't suit me.

I have a lot of Controllers and Actions using this pattern and I don't want to change them all. Calling ToList/ToArray also has a cost in terms of memory allocation and CPU usage. These may be minor and we shouldn't "prematurely optimise", but ignoring these issues felt sloppy. There's also the environment to consider. The API is running in a long lived instance of a client application, on low spec hardware and often in use by a local interactive user simultaneously. It wasn't designed up front to host the API, or to serve more than one user at once.  Memory fragmentation, garbage collections and excessive CPU usage would all be cause for alarm. In a perfect world we'd have a different design. At the moment time to market and available resources are choosing our path. Another concern is every developer who ever works on this has to avoid the same issue. This isn't obvious. The code compiles and runs fine on the server. It's only when you try to deserialise it in a remote process that you'll find a problem. Even then, you might not understand the problem or how to solve it. That moves the "pit of success" out of the common path.

I could have written a WebAPI filter which automatically called ToArray/ToList when appropriate. This would have solved some of the problems above, but not the efficiency issues.

There's always a question of "did you look hard enough?" but I only found one other published solution. This is to change the Action return type to IList<T>, List<T>, or Array<T> or something similar that is a 'more concrete' type. This has pretty much all the same issues as the other solution, perhaps more. Some people insisted this was more correct architecture anyway, which I question. It's possible Array<T> is more 'correct', but even then I think it's questionable. List/IList definitely doesn't seem better to me.

So without further ado, I present my solution: A JsonConverter for generic enumerators. This class plugs into the Json.Net serialiser, detects when the object to serialise is an IEnumerator<T> and writes out a Json array. Deserialisation is left to Json.Net to handle with its default logic. With this class and one line of code to use it, all my Controller Action methods now work as I expect.

Consider this code released under the MIT license, and with the caveat that it "works for me". I'm not aware of any immediate short comings, but I didn't code it for anyone else. There's no error handling, no unit tests. I haven't checked if it copes with nulls. I didn't worry about the naming. I haven't optimised the code for maximum performance. I didn’t try it with other generic types. It’s provided “as is” and I do not want to set any expectations of quality. YMMV.

You can configure the default serialiser to use it like this;

config.Formatters.JsonFormatter.SerializerSettings.Converters.Add(new LinqToIEnumerableJsonConverter());



Here's the class itself (also available as a gist);

internal class LinqToIEnumerableJsonConverter : JsonConverter
{
public override bool CanConvert(Type objectType)
{
if (!objectType.IsGenericType) return false;
var genericType = objectType.GenericTypeArguments.First();

var enumeratorType = typeof(IEnumerator<>).MakeGenericType(genericType);

return (objectType.GetInterface(enumeratorType.Name) != null);
}

public override bool CanRead
{
get
{
return false;
}
}

public override bool CanWrite
{
get
{
return true;
}
}

public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
return null;
}

public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
writer.WriteStartArray();
foreach (var item in (IEnumerable)value)
{
serializer.Serialize(writer, item);
}
writer.WriteEndArray();
}
}





Now go forth and serialise!

No comments:

Post a Comment