Lost In Tangent

I had a dream once where all developers had to care about was server code. We could just write services all day long without any regard to anyone actually using the functionality. It was a little lopsided and ridiculousness, but it made the job half as easy. Well, unfortunately we don’t live in that crazy fantasy world. Now that we’ve created this sweet data service, we have to actually create an application to consume the thing (absolute heresy!). Luckily for us, it’s actually really easy. In fact, I question whether it’s a little too easy. We all know that good code should be hard to read and so convoluted that not even The Da Vinci Code could decipher it. Maybe we can let our guard down just this once, and embrace the slim data service client API.

From previous posts, you’ll recall that because the data service has a RESTful interface, all we really need to work with it is a class that could make HTTP requests and give us the response. Which means if you really wanted to you could use the WebClient class like so:

Note: I truncated the URL parameter passed to DownloadString to conserve space. I’ll be doing a lot of this through the rest of the article.

You could then use LINQ To XML to parse the carriers string, and you would look like a coding god. You just used a class in the System.Net namespace, which is an automatic 100 cool points! Not so much. Treating the data service like a plain HTTP resource makes sense only in the minutest scenarios, and even then I don’t see why you would use the above code. When working with a data service, there are numerous additional features we need to make the whole process cleaner, that WebClient (or HttpWebRequest) can’t give us. I can think of at least four things I’d need out of a data service client:

1. Object deserialization
2. Lazy loading
3. Identity tracking (Identity Map for all you pattern nerds)
4. Change tracking (Unit of Work)

(Note: For anyone keeping track, I just covered all of Martin Fowler’s object-relational behavior patterns)

By “lazy loading”, I mean that we should be able to put off the retrieval of superfluous data until we actually need it. By “identity tracking” I mean that I need something that is smart enough to know what items I’ve already requested from the data service (based on identity), so that I only have to retrieve it once. By “change tracking” I mean that I need to be able to add/edit/delete items in the data service, and those changes (and only those changes) be tracked and persisted. #1 should be pretty self-explanatory. I don’t want to have to get a string of XML and have to parse that trash myself. When I request a Carrier from a data service, I want a Carrier object, not a <carrier> element.

All four of those tenets are satisfied by any worthwhile OR/M framework. LINQ To SQL’s DataContext handles that load, and the Entity Framework’s ObjectContext does too. To keep pure synergy across the entire suite of new ADO.NET tools, ADO.NET Data Services has been given the WebDataContext class. Not only does the name fit ([InsertWordHere]Context), but it contains the necessary functionality for abstracting/working with a data service, as well as providing the aforementioned features.

Creating a WebDataContext is as easy as:

“Where is my connection string!”, you might be saying. With data services, the URL of the service is the connection string. I don’t know why but that thought was a turning point for me in appreciating data services. The clearer it becomes that you’re able to interact with a data source just like you would using familiar/traditional methods (i.e. LINQ To SQL, EF), but over the wire, the more appealing this technology becomes.

So now that you’ve got your context, pointing at the right data service, how do you get data out of it? The WebDataContext class has a method called CreateQuery that takes a string representing the data query: entity name, and query parameters. For instance, if I was using the above context, and wanted to get all cell phones, I could simply do:

If I wanted to get all cell phones that have email, ordered by price, I could do:

Nothing new is going on here, and it isn’t necessarily impressing. At this point all we’ve really added is some semantic meaning to working with our data service that we didn’t have using WebClient or HttpWebRequest. That is definitely a benefit, but it isn’t incentive enough for it to be a save-all solution. We need to align the three tenets above to functionality of the WebDataContext for things to get really juicy.

1) Object Deserialization

This is absolutely the most important tenet (I suppose that’s actually a subjective statement). If you’re already used to using OR/Ms to streamline your data access, then you probably expect the same ease-of-use when working with data services. You may have noticed that the CreateQuery method is generic. It accepts the object type that represents the entity being requested. Then what happens is that the WebDataContext will attempt to serialize instances of the specified type from the data returned from the call to the data service, that way you can work with a collection of objects (or just one, depending on the query).

You may be asking, “Where on earth does that CellPhone class come from? This is the client after all, and that class was part of the LINQ To SQL model on the server.” And that would be a great question to ask. The cool thing is that the WebDataContext can perform the deserialization using pure POCO, which means that all your client application has to have is an object with the same name and same properties as the entity on the server, nothing else (there is an exception to this, that we’ll discuss later). This is great for simple types. For me to make requests for carriers I could simply declare the following type:

And then make the following call:

I now have a collection of Carrier objects, contained by a WebDataQuery object. Beautiful! We’ll discuss the benefits of the WebDataQuery class in the next post, for now it’s sufficient to note that it is just a client container for a data service query.

What is really cool about the above example is that for a client to consume a simple data service, they don’t need a bloated proxy. All you need is the URL of the service, and a plain object that matches that of the consumed entity. That kind of code cleanliness/simplicity makes me very happy

The first thought you might be having is: “I agree this is some great stuff, but what if I want to work with a data service that has multiple types, that are fairly complex? Hand coding those classes could get time-consuming.” You would be exactly right. The above example was meant to illustrate how basic working with a data service can be, but it wouldn’t always be practical to hand-code the client-side model. Luckily, ADO.NET Data Services comes with a tool to generate your client-side model. All you have to do is give it the URL of the data service and it will generate classes for each entity exposed by the service. It will also create a strongly-typed subclass of WebDataContext that provides nice properties for accessing each entity (just like LINQ To SQL and the Entity Framework). The tool is called webdatagen and is located in the Program Files\Microsoft ASP.NET 3.5 Extensions directory (this location is subject to change).

To illustrate it’s use, I’ll leverage webdatagen to create my client-model for the cell phone data service:

/outobjectlayer - This is the name of the file you want the generated classes to go in (the default language is C#, but you can set it to use VB)

/uri - Obviously this is the URL of the data service you want to generate the client model for

The webdatagen tool has a lot more options than this, but I’m not going to go into them. You can type /help to get all that info. The important thing is that after running the above command, I now have a file called CellPhoneModel.cs that contains classes for Carrier, Manufacturer, and CellPhone (those of the entities in that service), as well as a nice WebDataContext. I include the file in my project and can now run the following code:

That’s definitely a nicer alternative to what we were using before. Our code is now more explicit, and we’ve abstracted how we’re accessing the CellPhone data using the newly generated property. If you wanted to take it a step further, you could either create a default constructor on your generated WebDataContext that infers the service URL from somewhere (hard-coded, config file, etc) or you could use an IoC container to inject the URL into the existing constructor. Either way, you can see how easy working with a data service can be.

If you’re curious how the CellPhones property works, here is the generated code:

As you can see it’s the same exact logic we were running manually before, except in this case it’s also being smart enough to “cache” the CellPhone data, which we weren’t doing before.

2) Lazy Loading

Now that we are working with strongly-typed objects, we no doubt want to be able to access both scalar and relational properties, otherwise what’s the point of having a client-side model?. As you may recall from article #3, when you request an entity from a data service, its relationships are not initially loaded, which is a good thing, because you could potentially have a lot of relationships on an entity, that could result in a ton of unneeded data being transfered.

When making a request through a URL in a browser, or using the WebDataContext’s CreateQuery method, you can use the $expand query parameter to eagerly load any relationships. This approach is great, but also has a couple draw backs:

1. You have to know at the time the query is made that you want to expand a relationship (or relationships).
2. If you are querying an entire entity set, and you want to expand a relationship property, you have to expand it for every single entity in the collection. If you only want the relationship expanded for certain entities, based on some type of criteria, there is no way to do that, and you’ll end up transferring a potentially large amount of unneeded data.

Imagine I want to make a query for all cell phones. Easy right? In fact, we already saw that example done very simply in the previous section. Let’s revisit it for the sake of repetition…

That will return to us a collection of CellPhone objects, but their Carrier and Manufacturer properties won’t be populated (thanks to lazy loading). We decide that we also want to receive the cell phone’s manufacturer, so we modify the above code:

Perfect! We now have our manufacturer data. We later figure out that we only need the manufacturer data for phones that are less than $500 and weights more than 3 ounces (that is a ridiculous need, but work with me here). For us to get manufacturer data using the CreateQuery method, it’s all or nothing in regards to retrieving relationship data. We know we need to get some manufacturer data, but we’re really hesitant to just blindly pull down all manufacturers when we only need a few. It’s clear that CreateQuery doesn’t offer the functionality we need.

Luckily, CreateQuery isn’t the only method the WebDataContext class has to offer. LoadProperty allows you to specify an entity object instance, and the name of one of its relationship properties that you want to populate. LoadProperty is great because it’s an “after the fact” method. It allows you to query the base data you know you need up front, and then later pull down just the extra stuff, as you need it. In our current example, we want to query all cell phones (without expanding any relationships), and then programatically determine which cell phones need their manufacturer data, and at that point retrieve it. The following code would work beautifully:

“Isn’t that going to make a bunch of individual calls to the data service to retrieve the manufacturer data? We may have saved bandwidth up front by not expanding manufacturers, but now we’re going to pay the processing price by making a lot of extra calls to the service.” That is very true. The above example wasn’t meant to say that LoadProperty is the best solution every time, nor is making a broad query with CreateQuery the best solution every time. The point is, you have two options for performing your lazy loading, each of which provides a different level of precision with your data. If only 3 out of 1,200 cell phones meet the above criteria, then there is a good chance that using LoadProperty will be more performance. But if 639 out of 1,200 cell phones meet the criteria, you might want to weight the advantages between bandwidth and server calls.

You may be thinking to yourself right now, “You know, I’ve gotten really comfortable with the way LINQ To SQL handles lazy loading of relationship properties. All I have to do is access the property, and behind the scenes it pulls down the data I need. I don’t have to make any extra method calls, it just works. LoadProperty seems like an extra step to me.” You’d be very right. LINQ To SQL (and the Entity Framework for that matter) does make the process of lazy loading a little more streamlined. But, have you ever taken a look at how LINQ To SQL accomplishes its lazy loading? It uses three helper classes (EntityRef<T>, EntitySet<T>, Link<T>) to represent the lazy loaded properties, and relies on code in the property’s getter/setter to work with the underlying helper class reference. It isn’t hard to understand/develop, but it’s added code and added complexity. With ADO.NET Data Services, you don’t need any helper classes, and you don’t need any extra code in your properties. Unfortunately you don’t get the extra behavior, but you do have less dependencies in your entity classes, and less bloat.

You could easily hand-code an ADO.NET Data Services client-model, because it comes very close to being true POCO. The same couldn’t really be said about a LINQ To SQL model, and certainly not for an EDM. I personally love the leanness of the data service client model, and I think a call to LoadProperty is a fair trade-off. In fact, you could even look at a call to LoadProperty as adding a little more explicitness to your code, that you wouldn’t have by just accessing the property and having the plumbing take care of the magic for you.

3) Identity Tracking

So why is identity tracking so important exactly? Well as you can imagine, every time a client requests data from a service, it requires time to return the response, and it requires bandwidth from the server to satisfy the request. In the case of Phillip’s Cell Phone Shack, he is in the business of making money, and having to serve the same cell phone data over and over again, when unnecessary, isn’t the greatest idea ever conceived. If the client was stateless (web application) then such tracking wouldn’t hold as much merit, but if the client was a desktop application, you can imagine that being able to hold a local cache of entity data would save a lot of needless round trips.

Once again, just like LINQ To SQL’s DataContext, and the Entity Framework’s ObjectContext, the WebDataContext class has identity tracking built into it. If you request a cell phone object, and then request the same cell phone again, it will be smart enough to give you the existing instance and not make the service call. This is not groundbreaking functionality by any means, but it’s a requirement for efficient applications, and it’s great that it’s a part of the ADO.NET Data Services feature set.

Above, I mentioned that data services work with pure POCO, and that statement is true as you saw. I also mentioned that there was an exception. In order for the WebDataContext to perform identity tracking, it has to know how to determine the identity of each entity type it manages. In order for it to do that, each entity type has to abide by the same rules as the entity type on the server. The only difference is that instead of applying the WebDataKey attribute to your class, you apply the Key attribute. So if I wanted to enable tracking for my Carrier entity, I could simply do:

If you hand-coded your entity classes, and don’t care about identity tracking, then you’re fine. If you hand-coded your entity classes and you do care about identity tracking, then you’ll need to make sure your entity’s identity property is apparent to the WebDataContext. If you used the webdatagen tool, it assumes you care about identity tracking and applies the Key attribute to the entity type’s identifying property, if it needs to. I say “if it needs to” because as you remember, if the identity property is “ID” or “[EntityType]ID” then that will automatically work, just like the server-side model types. In our case, none of my entity types follow either of those conventions, so I need to explicitly use the Key attribute. If your entity type’s identifying properties do in fact follow one of those conventions, then your data service client could work with pure POCO and leverage identity tracking as well. How sweet is that

So now that your entity classes are identity tracking enabled, there is still one more step before you can actually benefit from identity tracking. Because identity tracking is an additional feature that may not be needed by the client application, it is turned off by default. In order to enable it, you have to set the WebDataContext’s MergeOption property to something other that NoTracking (the default value). The following code would enable identity tracking for our context:

By setting the MergeOption to AppendOnly, you’re telling the WebDataContext to only append new entity instances that it doesn’t already have, effectively turning on identity tracking. The MergeOption enumeration has two other values: OverwriteChanges, and PreserveChanges. These values relate to the change tracking features of the WebDataContext.

4) Change Tracking

The unit of work pattern is great because it allows you to make a bunch of changes in a single session and then submit all those changes for persistence in one swoop. This is by no means new, in fact .NET developers have enjoyed it for years. LINQ To SQL’s DataContext, Entity Framework’s ObjectContext, and even the DataSet, all represent implementation’s of the unit of work pattern. Thankfully, the WebDataContext is no different. It allows us to make many different changes to its entity data (add/update/delete) and then perform a batch save. Best of all it will only persist the data and entities that have been modified, which means no unnecessary data being sent across the wire and no unnecessary logic being performed.

“Woah woah woah!”, you may be saying. “What is all this jibber jabber about changing entity data? Can we even do that with a data service? I thought this sucker was read only? How do you even perform updates to a service that is RESTfully exposed anyways?”. Those are all good points. The cool thing is, yes, you can make changes to a data service’s entity data from a client application. This topic deserves an article in itself, and as such will be tabled until part 7 of this series. Fear not faithful reader, you will wield these great powers very soon. Just be patient and let me finish explaining the entire client-service querying capabilities gamut.

So we’ve added the WebDataContext class to our arsenal of tools. We’ve learned how to leverage the webdatagen tool to build our lean client-side model. We’ve seen how to benefit from lazy loading, identity and change tracking, as well as get full object deserialization. One problem still exists though: a richer coding model for querying the data service seems in order.

If you’ve gotten used to using LINQ, no doubt you might see the above code samples as feeling a little primitive in comparison. Well, today is your lucky day, because there is a new LINQ flavor in town, and it goes by the name LINQ To Data Services. In the next article, we will see how we can leverage our knowledge of consuming data services and throw it away. LINQ is in baby! Everything else is out!

Do you remember the day when you first started learning web development and you figured out what the stuff after the “?” was in the URL? And then you found out that was called a query string. Can you remember your excitement? Maybe you uttered the words “I’m going to use query strings forever!” Well, you probably didn’t say that exactly, but you definitely recognized the value of it.

Some of you might have been thinking while reading this article series “You know I’m loving all these RESTful service URLs, but I gotta say I’m really jonesin for a query string.” Well, your cravings end here. This article will introduce the notion of query parameters to data services, and explain what they can be used for.

ADO.NET Data Services take parameters in the form of a query string just like normal web pages. The only difference is that there is a set list of parameters you can specify, that following a specific convention: $[parametername]=expression. So if your query was /CellPhones and you were to apply a parameter to it, it would look like: /CellPhones?$[paramatername]=expression. Nothing new is going on here, you basically just have to note that data service parameters are prefixed by a dollar sign, otherwise it’s just a standard query string. I say “expression” instead of “value”, because as you’ll see, the parameters can take sophisticated expressions, complete with operators and functions.

So what parameters does the data service accept?

1. expand
2. orderby
3. skip
4. top
5. filter

Expand

Remember from the last post, when we queried a specific cell phone (the iPhone)? The returned content included two <link> elements that pointed at the phone’s carrier and manufacturer. That works great for situations where you’re not sure if you’re going to need that additional data (lazy loading). But what if you want both the cell phone and its carrier? This is where the expand parameter comes in. It allows you to specify the name of the relationship property that you want to automatically be retrieved (eager loading). Hence, we could use the following URL to satisfy our above requirement: /CellPhones(’iPhone’)?$expand=Carrier. Which would gives us the following (the returned content has been trimmed for brevity, but you’ll get the point):

Notice that we still have an empty <link> element for the Manufacturer relationship, but the Carrier <link> element has been expanded inline, as we requested. What if you wanted the Manufacturer as well? Luckily, the expand parameter can take a comma-delimited list of relationships to include, so /CellPhones(’iPhone’)?$expand=Carrier,Manufacturer would work just fine.

Orderby

By default the data service returns entity data ordered ascending by key (in our case, ModelName). In some cases this might be sufficient, but in a lot of situations it wouldn’t be. Suppose I wanted to retrieve all cell phones ordered by price. I could use the following URL: /CellPhones?$orderby=Price. If I wanted them ordered descending, I could simply use: /CellPhones?$orderby=Price desc. As you can imagine, there is also an “asc” keyword, but that is the default sort direction, so its use is limited. The orderby paramater takes a comma-delimited list just like expand, so you can build whatever order clause you need (i.e. /CellPhones?$orderby=Price desc,CarrierId,Zoom asc).

Skip

Pretty much all practical, real-world uses of data stores require paging, especially if the client application is on a different server. The skip parameter is 1/2 of the paging functionality that ADO.NET Data Services provides (the second half is discussed below). By applying a skip parameter, you are requesting that the data service skip a certain number of items, and then return all items after that. So if I were to request: /CellPhones?$skip=10, I would not receive the first 10 cell phones, but would get all remaining. The skip parameter determines which items to skip based on the sort expression for the data, so you want to make sure you also apply the orderby parameter if the default sort order isn’t what you want.

Top

If I only wanted to retrieve the top 10 most expensive phones, or the 5 lightest phones (or whatever), I could use the top paramater. You simply give it a number of items, and it will grab that many items off the top of the data source. Like skip, top also works in conjunction with the orderby, so you can imagine how you could use top to also simulate a bottom parameter. For instance, the following URL would retrieve the 10 most expensive phones: /CellPhones?$orderby=Price desc&top=10. This URL would retrieve the 5 lightest phones: /CellPhones?$orderby=Weight&top=5. I could then apply the skip paramater to begin paging: /CellPhones?$orderby=Weight&skip=5&top=5. When used together, the skip, top, and orderby parameters enable you to handle all your paging and reporting needs beautifully.

Filter

Now that we know how to retrieve an object’s entire graph, order, and page it, we start to realize the need for filtering some of this data. If we were looking to only retrieve cell phones for a specific manufacturer, we certainly wouldn’t want to have to pull them all down and do the filtering in our client code.

This is where the filter parameter comes in. It allows you to filter (big surprise) the entity data based on pretty much whatever criteria you want. You can even traverse relationships to filter an entity by a related entity’s properties. Remember when I mentioned that query parameters take expressions not values? That statement is most true for the filter parameter, which has a slew of acceptable operators and functions to handle your filter critieria.

The filter parameter has two types of operators: logical, and arithmetic. The logical operators are:

1. eq - Equal
2. ne - Not equal
3. gt - Greater than
4. gteq - Greater than or equal to
5. lt - Less than
6. lteq - Less than or equal to
7. and - Logical and
8. or - Logical or
9. not - Logical negation

The arithmetic operators are:

1. add - Addition
2. sub - Subtraction
3. mul - Multiplication
4. div - Division
5. mod - Modulus

The filter parameter also allows the use of parentheses to group expressions. I think the operators are pretty self-explanatory, and you can imagine how they allow you to build pretty complex expressions. For instance, suppose I wanted to get all phones that weight more than 3 ounces, have Bluetooth, have a camera with at least 2 megapixels, but don’t have email, and would be affordable if I had $500 to spend on a phone. I could do the following: /CellPhones?$filter=Weight gt 3 and Bluetooth eq true and Megapixels gteq 2 and Email eq false and 500 sub Price gteq 0. You get the point. The above 14 operators can be combined to do whatever filtering you’ll ever need.

Aside from the list of operators, the filter parameter also has 4 categories of functions:

1. String
2. Date
3. Math
4. Type

Each category is comprised of functions that make filtering on data of the respective type possible. For the sake of time I’m not going to go into every single function (there are over 30!), but you have things like: substring, toupper, indexof, round, ceiling, cast, etc. I might do a mini sub post just on the filter functions sometime in the future.

So now that we’ve mastered the basic form of querying ADO.NET Data Services, namely through a RESTful URL, how can we abstract this concept and work with a data service in a more concise fashion? The next article will focus on creating clients that consume data services.