Tales from the Evil Empire

Bertrand Le Roy's blog

News

Ads Via DevMavens

ASP.NET AJAX UpdatePanel Control: Add Ajax interactivity to your ASP.NET 2.0 web pages


follow bleroy at http://twitter.com


Add to Technorati Favorites

Blogs I read

My other stuff

Dates and JSON

JSON is a great data format and it's taken the Internet by storm for a number of good reasons. But because of a strange oversight in the EcmaScript specs, there is no standard way of describing dates in JSON. There's been a lot of discussion on this topic and it still remains a problem today.

In the Microsoft Ajax Library, we've tried a number of different approaches before we converged to the solution we're using today.

The first thing we tried was to inject Date constructors in the JSON string. This is a (very) bad idea for a number of reasons. First, it simply does not conform to the JSON specs. Second, any JSON parser that validates its input before parsing it will cough on such a thing. Finally, it establishes a precedent: why would it be allowed for dates and not for arbitrary types? This would just defeat the purpose of JSON.

The second approach, which is what most people settled on, is to adopt some string form of date and just interpret any string that conforms to this format as a date. We've been using "@1163531522089@" where the number is the number of milliseconds since January 1st 1970 UTC for a while (not super-readable) and some people are just using the ISO 8601 format. These are both almost acceptable compromises (if everyone agrees on them) but have a fundamental flaw which is that there can be false positives: what if you want to serialize the "1997-07-16T19:20:30.45+01:00" string, as a real string, not as a date?

To be perfectly honest, JSON Schema does solve the problem by making it possible to "subtype" a string as a date literal, but this is still work in progress and it will take time before any significant adoption is reached.

Our current approach is using a small loophole in the JSON specs. In a JSON string literal, you may (or may not) escape some characters. Among those characters, weirdly enough, there is the slash character ('/'). This is weird because there actually is no reason that I can think of why you'd want to do that. We've used it to our benefit to disambiguate a string from a date literal.

The new format is "\/Date(1198908717056)\/" where the number is again the number of milliseconds since January 1st 1970 UTC. I would gladly agree that this is still not super readable, which could be solved by using ISO 8601 instead.

The point is that this disambiguates a date literal from a string that looks like the same date literal, while remaining pure JSON that will be parsed by any standard JSON parser. Of course, a parser that doesn't know about this convention will just see a string, but parsers that do will be able to parse those as dates without a risk for false positives (except if the originating serializer escaped slashes, but I don't know of one that does).

Here's how you'd encode a string that contains the same literal I described above: "/Date(1198908717056)/".

Notice how a simple eval-based parser will just return the exact same string in both cases, but if you run a simple search for "\\/Date\((\d+)\)\\/" and replace with "new Date($1)" before the eval (but after validation), you'll get the dates right in the final object graph. The string I described above will just go through that filter undetected and will remain a string.

We're pretty much satisfied with this solution to the date problem, but of course for the moment very few serializers and parsers support that convention. It would be great if this could become the consensus across the industry.

I'd love to read any comments you may have on that subject.

Comments

Josh Stodola said:

>> It would be great if this could become the consensus across the industry. <<

I have been saying this about numerous Javascript inconsistensies for over 7 years. In that time span, I have yet to see a single consensus. The language is extremely desperate for standardization. It's about time somebody in authority puts their foot down! It seems that without a leader, it's going no where (and fast).

# January 18, 2008 4:20 PM

Bertrand Le Roy said:

Well, Josh, this is a little different in the sense that JSON (not directly talking about JavaScript date literals here) is in the hands of a few individuals, not a standards comittee so it may actually be achievable.

# January 18, 2008 4:24 PM

Joe Chung said:

One reason to escape slashes in JavaScript is that some browsers don't handle HTML close tags, e.g., </p>, well in JavaScript so you would use <\/p> instead.

# January 18, 2008 4:40 PM

Alex Osipov said:

For me, what gets confusing is DateTime serialization in regards to time zones.  .NET in general is not consistent when dealing with DateTime and time zones, so would love to see a more standard approach not just for Atlas but .NET in general.  An attribute allowing me to use local time on the client would be very helpful at times.  

# January 18, 2008 4:48 PM

Bertrand Le Roy said:

Thanks joe, that explains it.

Alex, do you have a specific example of such an inconsistency?

# January 18, 2008 4:50 PM

Kris Zyp said:

Bertrand, clever trick, and good job making it go with eval. However, I would still be hestitant to move away from the date serialization used by Crockford's library and ES4. I think your suggestion of defining a property as a date in a schema(JSON Schema or otherwise) outside of the actual data in order to explicitly declare dates, is a better technique, because the standard serialization/parsing libaries can still be used.

# January 18, 2008 5:30 PM

Bertrand Le Roy said:

I'll try to lure Doug to this post and get his thoughts. Good thing is that on the parser side, we can support several approaches. Again, Doug's convention will have false positives so most parsers will probably not do the special casing by default.

# January 18, 2008 5:36 PM

Nikhil Kothari said:

Actually, I should have updated my main post (a year ago I guess) but as discussed in the comments, there is a way to use the @ syntax without false positives.

Your post called my attention to it, and I updated it now. :-)

# January 18, 2008 10:54 PM

Rick Strahl said:

The question is who do we have to bribe to make *some* sort of standard for date literals :-}. This is a pretty big stumbling block in universal JSON usage, although the solution you came up with MS AJAX with the \/Date()\/ seems like a good way to go - it  works and is relatively easy to implement. I posted some additional thoughts and some code to that effect here a while back when we discussed this last month:

www.west-wind.com/.../214731.aspx

# January 19, 2008 8:01 AM

Michael Yeaney said:

I've worked around these problems in the past by using JSON to *only* transport formatted data - in other words, no reliance on the client browser native types at all for numbers and dates.  While it sounds a little odd, most of the enterprise products i've worked on supported the notion of a configurable timezone and/or culture, which makes the browsers cultural settings worthless.  So for dates and numbers, we do *all* formatting on the server (after setting the current thread culture code and format options as per the app config), and just ship the "ready for display" data to the browser.

IMO, until we can *force* the browser to use different time zone and number format options, it's pointless to do manipulation on the browser (and a *lot* more code to support - we tried it) - the .NET framework has most of these features built in (except for explicit DST support), so why not leverage them?

Any thoughts???

# January 19, 2008 12:31 PM

Eilon said:

Some languages out there already have Date literals - which ones are they? Why don't more languages have Date literals?

And for that matter, what are some other primitive types that lack literal support in languages?

# January 21, 2008 1:41 PM

rascunho » Blog Archive » links for 2008-01-21 said:

Pingback from  rascunho  &raquo; Blog Archive   &raquo; links for 2008-01-21

# January 21, 2008 3:24 PM

Wöchentliche Rundablage: .NET 3.5, WPF, LINQ, Tests, System.AddIn, SubSonic, Sandcastle | Code-Inside Blog said:

Pingback from  W&ouml;chentliche Rundablage: .NET 3.5, WPF, LINQ, Tests, System.AddIn, SubSonic, Sandcastle | Code-Inside Blog

# January 21, 2008 3:48 PM

Sweet said:

You guys are killin me. Why not put a few lines a code in here. Whatever you do make sure nobody can find this without spending an hour on google. Nice job

# February 12, 2008 5:38 PM

James Newton-King said:

I am really happy with how Json.NET 2.0 and LINQ to JSON is turning out. This release offers big improvements

# March 2, 2008 2:20 AM

Denny.NET said:

Passing a JSON object to a WCF service with jQuery

# March 3, 2008 8:55 PM

mike said:

Suggestion for your blog: each link for JSON is the same, I thought each link was to a relative article/source, so perhaps only link the first instance?

# April 22, 2008 4:59 PM

NZEYIMANA Emery Fabrice » Invalid JSON from NET - DateTime said:

Pingback from  NZEYIMANA Emery Fabrice &raquo; Invalid JSON from NET - DateTime

# May 14, 2008 2:21 PM

iron9light said:

Make it a string and use your own converter (or parser).

# September 9, 2008 2:46 AM

Bertrand Le Roy said:

@iron9light: that's assuming you know what is supposed to be dates in the data structure. That is sometimes true but not always.

# September 9, 2008 2:11 PM

Adam Peller said:

What does the RegExp match do to performance?  Quite often, it's going to be wasted cycles.

This problem has been around a long time, and I fear there is no elegant solution for JSON without a native Date type in JS, since it's basically a language subset.  If you do pass some date representation, though, you really ought to use some ISO subset.  It's readable, in the spirit of JSON, sortable, and has the ability to specify timezones or use local time.  Using Unix Epoch is going to make it very difficult to identify a date only without being off by one and getting different results somewhere in the world!

I guess a lot of this is moot now that the ES31 spec is out there and IE8 and FF3.1 seem to offer implementations.  The reviver/replacer arguments aren't as magical as something type-based, but it avoids the syntax problem and is extensible without impacting base performance.

# January 6, 2009 1:31 PM

Bertrand Le Roy said:

@Adam: no, RegExp matches do nothing to improve perf... About date literals in JavaScript, well, it's not going to happen as far as I know. We are looking at the native implementations that are now available, but I wouldn't say base performance is not impacted...

# January 6, 2009 2:32 PM

Adam Peller said:

@Bertrand: I guess all I meant to say was that under the new scheme, you would have the option of providing a reviver/replacer to trigger your code on a branch, as opposed to putting a RegExp on the main codepath for all JSON.  Having native implementations for JSON should also be a fairly big win, though I haven't seen any numbers yet to indicate how big.

# January 6, 2009 3:13 PM

Bertrand Le Roy said:

@Adam: yes, the native implementation in IE8 for example, is based on json2.js so it has the reviver/replacer infrastructure in place. We're exploring using that when it exists. But for a script implementation to have reviver/replacer, you need to traverse the deserialized graph, which, depending on the data, can be more expensive than the regular expression. I agree that the advantage is better extensibility and an easy way to opt in or out of most date encoding scheme.

# January 6, 2009 7:58 PM

Andrew Webb said:

Umm... I think I'm missing something here...

I have a "/Date(1198908717056)/" string in one of my vars.

Can you give me a simple line or two of JS that will convert this string to a JS date?  Thanks.

# March 25, 2009 12:56 PM

Bertrand Le Roy said:

@Andrew: it needs to be "\/Date(1198908717056)\/", not "/Date(1198908717056)/". Then calling (new Sys.Serialization.JavaScriptSerializer()).deserialize will convert it to a date object.

# March 25, 2009 5:01 PM

Andrew Webb said:

@Bertrand: That doesn't seem to work.  Perhaps you can help me.

The first thing to trip me up is that ExtJs's JSON decoder converts "\/Date(1238025600000)\/" (23 chars) to "/Date(1238025600000)/" (21 chars).

The second thing is that 'serialize' and 'deserialize' are static methods, so we don't want to construct JavaScriptSerializer.

So I did this simple test:-

var s = "\\/Date(1238025600000)\\/";

var today = Sys.Serialization.JavaScriptSerializer.deserialize (s);

Before calling the deserialize method I checked that the contents of 's' were correct, and that its length was 23.  's' contains "\/Date(1238025600000)\/", trust me!  So now it should be able to be deserialized right?

However the 'deserialize' method throws this error:-

"Microsoft JScript runtime error: Sys.ArgumentException: Cannot deserialize. The data does not correspond to valid JSON.

Parameter name: data"

The correct technique still eludes me.  I searched high and low on the Internet yesterday and found loads of implementations - lengthy and short.  I tried a few; none of them worked!

What is the correct way please?

TIA

Andrew

P.S. I tried enabling script globalization on the ScriptManager - it didn't help.  I'm using ASP.NET 3.5 SP1, btw.

# March 26, 2009 5:29 AM

Bertrand Le Roy said:

@Andrew: I see. Sooo...

* ExtJS has no idea that this convention exists so I'm not surprised this doesn't work.

* Apologies about the static method. Of course it is. My mistake.

* If you want this to be valid JSON, it has to be a string within a JSON-valid string. This is because the actual common use case is '{foo: "\/Date(1238025600000)\/"}' (omitting escaping here). So if you just want the date on its own, unwrapped, you probably need to do '"\/Date(1238025600000)\/"'.

# March 26, 2009 2:04 PM

Andrew Webb said:

@Bertand: ah, I see.  My mistake.  Thanks.  So now...

var msJsonDate = '{ data: "\\/Date(1238025600000)\\/" }';

var day = Sys.Serialization.JavaScriptSerializer.deserialize(msJsonDate);

alert(day.toString());

This works, gives me a date, although the alert shows "[object Object]".  So it's not a fully functional Date object.

After more experimentation, I decided on this:-

 var s = "\\/Date(1238025600000)\\/";

 s = s.slice(7, 20);

 var n = parseInt(s);

 var d = new Date(n);

 alert(d.toString());

This alert shows "Thur Mar 26 00:00:00 UTC 2009".

# March 27, 2009 3:41 AM

Bertrand Le Roy said:

@Andrew: Sure, you need to do day.data.toString(), not day.toString.

When I do this: alert(Sys.Serialization.JavaScriptSerializer.deserialize('"\\/Date(1238025600000)\\/"').toString()); I am getting "Wed Mar 25 17:00:00 PDT 2009" as expected.

Please do not do your own parsing.

# March 27, 2009 4:02 AM

Andrew Webb said:

@Bertrand:

* "Sure, you need to do day.data.toString(), not day.toString" - of course; silly me.

* '"\\/Date(1238025600000)\\/"' - please explain this double-stringing technique.  Coming from a C, C++, C# background, I've never seen this!

* "Please do not do your own parsing" - fair play.  If I use 'deserialize' then you can change the format in the future, and my JS will still work.  I get it.

# March 27, 2009 2:03 PM

Bertrand Le Roy said:

@Andrew: for the double string, it looks artificial in this trivial case but it's actually quite simple. The deserialize method takes a string as its parameter. This string /contents/ are the JSON representation of the object to rehydrate (which can be the representation of a string, a number, a Boolean, an object, etc., the deserializer doesn't know in advance, nor does it have to).

Now let's imagine we are deserializing a string object containing 42. In other words the string "42". In JSON notation, it is represented by: "42" (and not: 42). Now if you want to express that JSON string as a JavaScript string constant, this would be var a = '"42"'; and not var a = '42';. The first one will be deserialized as the string containing 42 as expected, whereas the second one will be deserialized into the number 42. I hope this helps clarify.

# March 27, 2009 5:26 PM

Andrew Webb said:

@Bertrand: thanks for taking the time to explain this to me, and for your other answers.  I feel a bit like a fish out of water when doing JavaScript, but I'm slowly getting there.  Crockford's "JavaScript: The Good Parts" is my guide; I've read it once, and it's probably time to read it again.  Thanks again.

# March 29, 2009 10:48 AM

David Farrugia said:

I disagree that the introduction of "new Date(millis)" to express a date is as bad an idea as you suggest.

Understood strictly, the JSON spec doesn't provide for passing dates - end of discussion. Any solution involving parsing strings, revivers, replacers and the like is fine, but is a separate concern and should no more infect the JSON spec as it stands than should the calling of arbitary constructors. The slash-escaping idea just looks like a terrifying hack.

Why not accept this, and define an "extended-JSON" which allows the contruction of Date objects, using the unambiguous number argument, if you consider Date objects to be fundamental?

That is to say, either dates are fundamental, and you wish to serialize them without scope for error, or they are not, and they should stay out of the data format. I don't see that it sets a precedent.

# May 14, 2009 4:35 PM

FG said:

I'm getting this format:

/Date(1233323754523+0100)/

I assume the +0100 is the timezone ? Not mentioned in your article.

# May 16, 2009 4:20 AM
Leave a Comment

(required) 

(required) 

(optional)

(required)