Dates and JSON

JSON is a great data format and it's taken the Internet by storm for a number of good reasons. But because of a strange oversight in the EcmaScript specs, there is no standard way of describing dates in JSON. There's been a lot of discussion on this topic and it still remains a problem today.

In the Microsoft Ajax Library, we've tried a number of different approaches before we converged to the solution we're using today.

The first thing we tried was to inject Date constructors in the JSON string. This is a (very) bad idea for a number of reasons. First, it simply does not conform to the JSON specs. Second, any JSON parser that validates its input before parsing it will cough on such a thing. Finally, it establishes a precedent: why would it be allowed for dates and not for arbitrary types? This would just defeat the purpose of JSON.

The second approach, which is what most people settled on, is to adopt some string form of date and just interpret any string that conforms to this format as a date. We've been using "@1163531522089@" where the number is the number of milliseconds since January 1st 1970 UTC for a while (not super-readable) and some people are just using the ISO 8601 format. These are both almost acceptable compromises (if everyone agrees on them) but have a fundamental flaw which is that there can be false positives: what if you want to serialize the "1997-07-16T19:20:30.45+01:00" string, as a real string, not as a date?

To be perfectly honest, JSON Schema does solve the problem by making it possible to "subtype" a string as a date literal, but this is still work in progress and it will take time before any significant adoption is reached.

Our current approach is using a small loophole in the JSON specs. In a JSON string literal, you may (or may not) escape some characters. Among those characters, weirdly enough, there is the slash character ('/'). This is weird because there actually is no reason that I can think of why you'd want to do that. We've used it to our benefit to disambiguate a string from a date literal.

The new format is "\/Date(1198908717056)\/" where the number is again the number of milliseconds since January 1st 1970 UTC. I would gladly agree that this is still not super readable, which could be solved by using ISO 8601 instead.

The point is that this disambiguates a date literal from a string that looks like the same date literal, while remaining pure JSON that will be parsed by any standard JSON parser. Of course, a parser that doesn't know about this convention will just see a string, but parsers that do will be able to parse those as dates without a risk for false positives (except if the originating serializer escaped slashes, but I don't know of one that does).

Here's how you'd encode a string that contains the same literal I described above: "/Date(1198908717056)/".

Notice how a simple eval-based parser will just return the exact same string in both cases, but if you run a simple search for "\\/Date\((-?\d+)\)\\/" and replace with "new Date($1)" before the eval (but after validation), you'll get the dates right in the final object graph. The string I described above will just go through that filter undetected and will remain a string.

We're pretty much satisfied with this solution to the date problem, but of course for the moment very few serializers and parsers support that convention. It would be great if this could become the consensus across the industry.

I'd love to read any comments you may have on that subject.

UPDATE: the regular expression can now find dates before 1970. Thanks to Rick and Marc for the tip.

61 Comments

  • >> It would be great if this could become the consensus across the industry. <<

    I have been saying this about numerous Javascript inconsistensies for over 7 years. In that time span, I have yet to see a single consensus. The language is extremely desperate for standardization. It's about time somebody in authority puts their foot down! It seems that without a leader, it's going no where (and fast).

  • Well, Josh, this is a little different in the sense that JSON (not directly talking about JavaScript date literals here) is in the hands of a few individuals, not a standards comittee so it may actually be achievable.

  • One reason to escape slashes in JavaScript is that some browsers don't handle HTML close tags, e.g., &lt;/p&gt;, well in JavaScript so you would use &lt;\/p&gt; instead.

  • For me, what gets confusing is DateTime serialization in regards to time zones. .NET in general is not consistent when dealing with DateTime and time zones, so would love to see a more standard approach not just for Atlas but .NET in general. An attribute allowing me to use local time on the client would be very helpful at times.

  • Thanks joe, that explains it.

    Alex, do you have a specific example of such an inconsistency?

  • I'll try to lure Doug to this post and get his thoughts. Good thing is that on the parser side, we can support several approaches. Again, Doug's convention will have false positives so most parsers will probably not do the special casing by default.

  • Actually, I should have updated my main post (a year ago I guess) but as discussed in the comments, there is a way to use the @ syntax without false positives.

    Your post called my attention to it, and I updated it now. :-)

  • I've worked around these problems in the past by using JSON to *only* transport formatted data - in other words, no reliance on the client browser native types at all for numbers and dates. While it sounds a little odd, most of the enterprise products i've worked on supported the notion of a configurable timezone and/or culture, which makes the browsers cultural settings worthless. So for dates and numbers, we do *all* formatting on the server (after setting the current thread culture code and format options as per the app config), and just ship the "ready for display" data to the browser.

    IMO, until we can *force* the browser to use different time zone and number format options, it's pointless to do manipulation on the browser (and a *lot* more code to support - we tried it) - the .NET framework has most of these features built in (except for explicit DST support), so why not leverage them?

    Any thoughts???

  • Some languages out there already have Date literals - which ones are they? Why don't more languages have Date literals?

    And for that matter, what are some other primitive types that lack literal support in languages?

  • You guys are killin me. Why not put a few lines a code in here. Whatever you do make sure nobody can find this without spending an hour on google. Nice job

  • Make it a string and use your own converter (or parser).

  • @iron9light: that's assuming you know what is supposed to be dates in the data structure. That is sometimes true but not always.

  • What does the RegExp match do to performance? Quite often, it's going to be wasted cycles.

    This problem has been around a long time, and I fear there is no elegant solution for JSON without a native Date type in JS, since it's basically a language subset. If you do pass some date representation, though, you really ought to use some ISO subset. It's readable, in the spirit of JSON, sortable, and has the ability to specify timezones or use local time. Using Unix Epoch is going to make it very difficult to identify a date only without being off by one and getting different results somewhere in the world!

    I guess a lot of this is moot now that the ES31 spec is out there and IE8 and FF3.1 seem to offer implementations. The reviver/replacer arguments aren't as magical as something type-based, but it avoids the syntax problem and is extensible without impacting base performance.

  • @Adam: no, RegExp matches do nothing to improve perf... About date literals in JavaScript, well, it's not going to happen as far as I know. We are looking at the native implementations that are now available, but I wouldn't say base performance is not impacted...

  • @Bertrand: I guess all I meant to say was that under the new scheme, you would have the option of providing a reviver/replacer to trigger your code on a branch, as opposed to putting a RegExp on the main codepath for all JSON. Having native implementations for JSON should also be a fairly big win, though I haven't seen any numbers yet to indicate how big.

  • @Adam: yes, the native implementation in IE8 for example, is based on json2.js so it has the reviver/replacer infrastructure in place. We're exploring using that when it exists. But for a script implementation to have reviver/replacer, you need to traverse the deserialized graph, which, depending on the data, can be more expensive than the regular expression. I agree that the advantage is better extensibility and an easy way to opt in or out of most date encoding scheme.

  • Umm... I think I'm missing something here...

    I have a "/Date(1198908717056)/" string in one of my vars.

    Can you give me a simple line or two of JS that will convert this string to a JS date? Thanks.

  • @Andrew: it needs to be "\/Date(1198908717056)\/", not "/Date(1198908717056)/". Then calling (new Sys.Serialization.JavaScriptSerializer()).deserialize will convert it to a date object.

  • @Bertrand: That doesn't seem to work. Perhaps you can help me.

    The first thing to trip me up is that ExtJs's JSON decoder converts "\/Date(1238025600000)\/" (23 chars) to "/Date(1238025600000)/" (21 chars).

    The second thing is that 'serialize' and 'deserialize' are static methods, so we don't want to construct JavaScriptSerializer.

    So I did this simple test:-

    var s = "\\/Date(1238025600000)\\/";
    var today = Sys.Serialization.JavaScriptSerializer.deserialize (s);

    Before calling the deserialize method I checked that the contents of 's' were correct, and that its length was 23. 's' contains "\/Date(1238025600000)\/", trust me! So now it should be able to be deserialized right?

    However the 'deserialize' method throws this error:-

    "Microsoft JScript runtime error: Sys.ArgumentException: Cannot deserialize. The data does not correspond to valid JSON.
    Parameter name: data"

    The correct technique still eludes me. I searched high and low on the Internet yesterday and found loads of implementations - lengthy and short. I tried a few; none of them worked!

    What is the correct way please?

    TIA

    Andrew

    P.S. I tried enabling script globalization on the ScriptManager - it didn't help. I'm using ASP.NET 3.5 SP1, btw.

  • @Andrew: I see. Sooo...
    * ExtJS has no idea that this convention exists so I'm not surprised this doesn't work.
    * Apologies about the static method. Of course it is. My mistake.
    * If you want this to be valid JSON, it has to be a string within a JSON-valid string. This is because the actual common use case is '{foo: "\/Date(1238025600000)\/"}' (omitting escaping here). So if you just want the date on its own, unwrapped, you probably need to do '"\/Date(1238025600000)\/"'.

  • @Bertand: ah, I see. My mistake. Thanks. So now...

    var msJsonDate = '{ data: "\\/Date(1238025600000)\\/" }';
    var day = Sys.Serialization.JavaScriptSerializer.deserialize(msJsonDate);
    alert(day.toString());

    This works, gives me a date, although the alert shows "[object Object]". So it's not a fully functional Date object.

    After more experimentation, I decided on this:-

    var s = "\\/Date(1238025600000)\\/";
    s = s.slice(7, 20);
    var n = parseInt(s);
    var d = new Date(n);
    alert(d.toString());

    This alert shows "Thur Mar 26 00:00:00 UTC 2009".

  • @Andrew: Sure, you need to do day.data.toString(), not day.toString.
    When I do this: alert(Sys.Serialization.JavaScriptSerializer.deserialize('"\\/Date(1238025600000)\\/"').toString()); I am getting "Wed Mar 25 17:00:00 PDT 2009" as expected.
    Please do not do your own parsing.

  • @Bertrand:

    * "Sure, you need to do day.data.toString(), not day.toString" - of course; silly me.

    * '"\\/Date(1238025600000)\\/"' - please explain this double-stringing technique. Coming from a C, C++, C# background, I've never seen this!

    * "Please do not do your own parsing" - fair play. If I use 'deserialize' then you can change the format in the future, and my JS will still work. I get it.

  • @Andrew: for the double string, it looks artificial in this trivial case but it's actually quite simple. The deserialize method takes a string as its parameter. This string /contents/ are the JSON representation of the object to rehydrate (which can be the representation of a string, a number, a Boolean, an object, etc., the deserializer doesn't know in advance, nor does it have to).
    Now let's imagine we are deserializing a string object containing 42. In other words the string "42". In JSON notation, it is represented by: "42" (and not: 42). Now if you want to express that JSON string as a JavaScript string constant, this would be var a = '"42"'; and not var a = '42';. The first one will be deserialized as the string containing 42 as expected, whereas the second one will be deserialized into the number 42. I hope this helps clarify.

  • @Bertrand: thanks for taking the time to explain this to me, and for your other answers. I feel a bit like a fish out of water when doing JavaScript, but I'm slowly getting there. Crockford's "JavaScript: The Good Parts" is my guide; I've read it once, and it's probably time to read it again. Thanks again.

  • I'm getting this format:
    /Date(1233323754523+0100)/

    I assume the +0100 is the timezone ? Not mentioned in your article.

  • Try this.

    var date_string = '/Date(1233323754523+0100)/'
    var d = new Date(parseInt(/\/Date\((\d+).*/.exec(date_string)[1]))

  • You can use -
    eval('new' + dateStr.replace(/\//g, ' '));

    this returns a Date and then use format function format the date as desired

  • @Neeta, &nbsp;That just ignores the timezone adjuster +0100. &nbsp;Be careful with daylight savings time if you are using a server in a different timezone to the target country of your site. &nbsp;I use a 3rd party JSON API which is hosted in GMT-700 SF, USA. &nbsp;However the target audiance is worldwide and I am testing in UK. &nbsp;Dates I added earlier in the year during BST (British Summer Time) but before SF went back an hour were an hour out though the time zone adjuster always showed -700!

  • Does Sys.Serialization.JavaScriptSerializer.deserialize() account for the timezone adjuster?

  • @Piyush: unfortunately, there is no direct support for that in JavaScript or in the Microsoft Ajax library. One thing you can do is add the offset yourself, like this: date.setTime(date.getTime() + timeZoneOffset) where timeZoneOffset would be 5 hours and a half.

  • If you need the date parts on the client, why not pass down the date as a json object. The only drawback I can see is verbosity. But how many dates do you need to return on json format where you need more than the text?

    [
    {"Date":{"y":2010,"M":05,"d":23,"H":12,"m":11,"s":58}, "DateString":"May 23, 2010 12:11 PM"}
    {"Date":{"y":2030,"M":4,"d":25,"H":16,"m":28,"s":32}, "DateString":"April 25, 2030 04:28 PM"}
    ]

  • What a horrific solution. It's completely unreadable to humans. Why don't we just serialize everything to binary if you're going to take that approach?

    An obvious solution would have been to work with ISO 8601.

    "\/Date(2010-01-17 14:00:00)\/"

    Something along those lines.

  • I just wanted to comment that the regular expression "\\/Date\((\d+)\)\\/" does not work for an example such as "/Date(-1233323754523)/". It doesn't pick up the negative. To fix this use: "^/Date\\((-?\\d+)\\)/$"

  • Seriously? All this to solve one single use case that will occur in, what? One in a hundred apps? One in a thousand?

    JSON is not meant to be a self-describing format. You have XML for that. If you want types, if you want differentiation, use that. Don't try to shoehorn a square peg in a round hole. Get a round peg.

    As it stands, this "solution" will require parsers everywhere to either be aware that they're talking to a Microsoft-based provider (breaking encapsulation and platform agnosticism) or waste resources testing for this format, rather than using a single, clear, platform agnostic, standardized format like ISO.

    This is the embodiment of a negative feature and a solution to a problem no one's having. Thank you for making my code more complex to solve a non-existent scenario. And when developers complain about Microsoft's approach to interoperability, THIS is exactly what they're complaining about.

  • @Julien:
    * even with your made-up statistics, one in a thousand, on a platform that is used by millions of developers, that is still thousands.
    * JSON has support for all JavaScript types, except date. It's not by design, it's an oversight in the JavaScript spec. If JSON really wasn't "self-describing" like you claim, we'd be writing "SomeNumber": "6". What is by design is the lack of support for types of complex objects, not the lack of support for dates.
    * This solution does not require parsers to be aware of that feature of the Microsoft serializers: if they aren't they are just going to see a string, the same way that if you're using ISO dates, you are really using strings. It also doesn't prevent anyone from using ISO dates, but if you do, you need to know in advance that this or that property is a date. Without that knowledge you can't differentiate between "4/6/2010" and 4/6/2010.

  • @Julien: the use case is when the consumer is a generic data consumer that doesn't have prior knowledge of a specific schema and that's either editing or processing the raw data. But you are absolutely right that the proportion of such applications may be small when compared with those where knowledge of the schema exists in the consumer. You're also correct that producers mindlessly exposing dates in that format can incur some cost on their customers. Ideally, I would have preferred the default to be ISO and this to have been an option. The .NET parser does understand both formats though.
    You cannot really say that JSON is only inspired from JavaScript notation. It means, after all "JavaScript Object Notation" and there is nothing in it that can't be parsed with a single eval. It does not have date expressions because JavaScript doesn't have a syntax for date literals. Date is not a complex type: it is not a composition of strings, numbers and Booleans. At least not any more than strings are sequences of numbers or numbers sequences of Booleans. It's a fundamental type that is present in all languages.
    The weakness of JSON and more generally JavaScript with regards to dates (lack of proper globalization for example) has been and continues to be a pain point for the industry, not a design strength.
    By the way, Crockford does object to this extension as well, but from what I understand not because there's no legitimate need for dates in JSON but more because it fragments the spec. I would today tend to agree with that as I said earlier.
    Good luck with the code. Note that WCF has added a time zone specification to their date literals but it remains fairly easy to parse: look for \/ at the beginning and end of string literals, and if found extract the integer and time zone shift. The integer is the number of milliseconds since January 1st 1970 UTC, which is something that many languages and class libraries accept as arguments to construct a date. Add the zone shift as hours and you should have your date object.

  • Bertrand,

    It would be good if updated your original post to use the regular expression from Rick Burgstaler's comment so that dates before January 1 1970 (where the numeric value is negative) can be parsed as well:

    "\\/Date\((-?\d+)\)\\/"

    I just got bitten by this in an app I developed some time ago.

  • @Marc: thanks!

  • Let's assume the TXT variable defined and initialized as follows:

    var txt = "{\"UpperLimit\": \"\\/Date(1290117600000)\\/\"}";

    Then the following code (a *safe* deserialization) will throw an exception:

    var deco1 = Sys.Serialization.JavaScriptSerializer.deserialize(txt, true);

    But the following code (a *regular* deserialization) will work successefuly:

    var deco2 = Sys.Serialization.JavaScriptSerializer.deserialize(txt);


    Any ideas?
    Thank you in advance!

  • @Matanel: that might be a bug. I'll contact the current owner of the feature. You may have to do your own sanitization if the payload comes from an untrusted source.

  • Ok, a simple question in plain English from someone who isnt much of a programmer but has to import a lot of JSON into a database. I have got an embedded date in that JSON and its format is:
    "date_created":"\/Date(1293595692063-0700)\/"

    How do I get this into a MM/DD/YYYY HH:MM:SS format (in my local timezone). I am in Eastern Timezone. I use VBA in MS Access 2010 but could get VB 2010 Express if need be.

  • The -0700 at the end is giving you the time zone. The number before that is the number of milliseconds since 1/1/1970. Most languages have an easy way to transform that into a date.

  • I'm curious; you provide a regex example to parse this string but later say, "don't do your own parsing". It seems that for those of us without access to the Sys namespace in JS, it would be better to parse for the string inside the parens, split on the hyphen, get a date object from arr[0] and calculate the tz differential on that result with arr[1] (if arr.length>1). Or am I missing something?

  • @Brett: feel free to do that if you want to.

  • Hi, i m having same problem using extjs which expects json data and bind it with grid.
    var output = serializer.Serialize(oOffer).Replace("\"\\/Date(", "new Date(").Replace(")\\/\"", ")");
    i tried this at server side but the result is complete UTC date time which i dont want i tried to ad fromat after new Date().format('Y-m-d') but still not working can some help me regarding this issue

  • Sorry to see that more of this didn't focus on human readability. &nbsp;Why do we keep having the same coversations over and over again? &nbsp;Because we loose track of the real issues and focus on the technology. &nbsp;As has already been pointed out, if Human readability is not the goal, why use JSON at all? &nbsp;Why not just send bits?

  • So is there anyway to specify that I want the DateTime to serialize to the iso format?

    Thanks

  • so, how would you display a format of a date such as &quot;12/14/2011 3:44:38 PM&quot; &nbsp; Mine just comes out as: &nbsp;&quot;/Date(1323907002367)/&quot;. &nbsp;shouldn't there be a function to use? &nbsp;Why can I not find it on Jquery.com? &nbsp;Why is there only a datepicker function? &nbsp;this long thread doesn't even have an answer, so someone post it.

    Thanks.

  • This is actually not a good way to encode dates because its based on a specific point in time. e.g. i want to encode the year 1000... now what do i do if we using millis since 1970?

    1) String format is much more flexible, transparent and portable.
    2) there is no need to indicate it as a date because the parser already knows if it expects a date. You don't get your error any earlier using an escape.

    The format pretty much needs to be a string and even better that it be human readable.

    I actually don't see the need to disambiguate the value at all... if you are expecting a date, then you already know what the format is, otherwise it is a string that still looks like a date.
    No problem.
    This works for programatic conversions as well in both typed and non-typed languages and generates an error at roughly the same time. You gain nothing by indicating adding type to the data.

  • @Brill: please read about JavaScript dates and you'll see that 1/1/1970 *is* the reference date that JS takes to define dates, and not something we arbitrarily decided on. You may use negative numbers.
    You are wrong that the parser knows where to expect a date. It simply doesn't and that's precisely the issue. Dates are the only simple type where this ambiguity exists. You are just missing the point.

  • A few hours ago I came into a bump with .NET JsonSerializer and Javscript.
    I haven't found a discussion about this on the web so here's my little sand grain :).

    After a couple of hours of hair pulling I found that DateTime objects where being serialized as UTC by .NET (something not quite nice when dealing with timezones and datetimes are critical).
    But that was not all. Javscript does its own conversion also. When running new Date() I found that JS was converting those milliseconds to the client's timezone (also, something not nice when having to render the date/time in another timezone than the one the user it sitting at).

    Hope this come in handy.

  • Holy smoke!

    Transferring data (POJO - so I do know what's being transferred) between Spring server and Java client was quite easy to get working with Json. Dates, sets, everything. Jackson library on both ends.
    When I tried that same interface with .net -> kaboom!
    What heavens have MS people thought, when their Json libraries cannot decode other than "Date\/(..)\/" format? Thrown normal milliseconds-from-epoch number to it and cry cry not correct format. Does that suck or what?

    What worries me now is that if the same server was made with asp.net MVC, it would stuff every time and date with this "victory day" nonsense. Zero compability, I would call it.

  • @Kimmy: super-easy to work around. For example, read Dave Ward's post on this: http://encosia.com/how-i-handle-json-dates-returned-by-aspnet-ajax/

  • @Bertrand:
    Sorry, I'm not sure where the answer in that link should have been, but it seems that custom datamember conversion did the trick for format (sorry about the code:

    [IgnoreDataMember]
    public DateTime startdate { get; set; }
    [DataMember(Name="startdate")]
    private UInt64? _startdate
    {
    get
    {
    if (this.startdate.CompareTo(unixEpochEET) > 0)
    return (UInt64)(this.startdate - unixEpochEET).TotalMilliseconds;
    else return null;
    }
    set
    {
    this.startdate = new DateTime(unixEpochEET.AddMilliseconds(Convert.ToUInt64(value)).Ticks, DateTimeKind.Unspecified);
    }
    }

    This way the date within the object knows how it should be got and set. I guess that I can use this method also in server side for communicating with non-ASP.NET or non-Dot.NET clients?

    The other thing is timezones... Jackson Json converts dates to GMT without asking (and without possibility to disable that behaviour it seems) which in my case sets all dates 2 hours off. For now I did dirty correction - use epoch+2 as base in Dot.NET client.
    My case might be special in that I don't want timezoned times: everything that is entered or read, is absolute time - it can't vary depending on where the reader is. Maybe ISO formatted string would be the
    ultimate date format for me, but I don't know if database can convert it to date/time automatically ;)

    So nothing wrong with ASP.NET Json as long there's a way to avoid it's formatting ;)

  • This format for date-strings is (at least in this example) specific to JSON being serialized using Microsoft's JavaScript libraries, correct? They look for and handle these date-strings?

  • Unless I'm missing something this &quot;solution&quot; depends on the exact string value being preserved, which is not guaranteed to be the case for a conforming JSON implementation.

    Or am I missing something here?

  • @Julian: not "sure" what you "mean" by "not guaranteed to be the case for a conforming JSON implementation". In any case, everybody's moving away from this and using ISO strings despite their disadvantages. Nothing left to see here. Move along.

  • Despite that fact that eval is dangerous, it is also Javascript specific. Not everything that consumes JSON is written in Javascript.

  • I know it's too late, and sorry to be negative, but in my opinion this is useless over-engineering, and more hassle than good.

    JSON is (mostly) untyped. Deal with it. That's what makes it good at bridging the gap between different languages (the fact that Javascript has a special type for datetimes is mostly irrelevant here: imagine Javascript had a special type for shoe sizes: would it be wise to enshrine *that* in an echange format designed to bridge the gaps between languages? I think not).

    This is a mistake. It reminds me of another mistake (which we also owe to Microsoft) of putting a BOM at the front of an UTF-8 file.

  • It's more than late, it's come and gone, but thanks for your opinion. Of course JSON is typed: it has numbers, strings, booleans, arrays, etc. The lack of date literals still is a gross oversight in the design of the language: it's the only basic type that doesn't have literal syntax. Even regular expressions have one. Your example of shoe sizes is absurd: dates are a basic type in all languages, like numbers.

Comments have been disabled for this content.