dive into mark

Microsoft.com web services gateway. A reformulation of Microsoft.com web services as a REST API. The original service posted SOAP messages and received SOAP responses; our gateway service uses pure HTTP GET for requests and pure XML for responses. Go read all about it, then come back and read on.

Several interesting things to note here:

• Both services support the same functions and accomplish the same things. In fact, the gateway doesn’t know anything about the functions it defines; it simply converts all the information into a SOAP message and calls the original service on Microsoft’s server, then munges the response and returns it.
• Both services use the Web Services Security UsernameToken Profile (PDF) algorithm for securely passing the developer token and secret PIN. The REST version emits a proper HTTP 401 challenge if authentication information is absent, and then expects the actual information in a custom HTTP header (X-WSSE). This is similar to the technique we used to extend HTTP authentication in the Atom API. Microsoft’s original version ignores HTTP’s native authentication mechanisms and reimplements everything in SOAP. (For the purposes of this demonstration, the gateway server will generate default authentication information if none is given. The sample Python client shows how to pass your own, if you signed up for your own developer token Thanks to Sam for pointing out that not mentioning this was confusing.)
• Although not shown in the examples below, the REST version supports Etags and gzip compression. The sample Python client shows how to take advantage of these basic features of the web that Microsoft’s SOAP server ignores.
• Another non-obvious benefit of the REST version is that standard Apache access logs will show you the actual usage of the service, since the URL contains both the method name and the function arguments. Access logs on the original service will only show the URL, which is relatively unhelpful since all requests are POSTed to a single URL; the method name is stored in a non-logged custom SOAPAction header, and the function arguments are stored in the non-logged SOAP body. (The body part of the body is the body, and the headers part of the body is the headers. But not all of the headers can go in the body. Just the authentication headers.) In other words, Microsoft would need to write custom SOAP-specific logging software to get all the logging that we get for free.
• The REST version uses less bandwidth, even if you’re not using Etags or gzip compression. There’s simply less stuff going back and forth between the client and server.

Here’s an apples-to-apples comparison of the wire formats.

To get the version of the web service, you send this to ws.microsoft.com:

Host: ws.microsoft.com
POST /mscomservice/mscom.asmx HTTP/1.0
Content-type: text/xml
SOAPAction: "http://www.microsoft.com/GetVersion"

<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
  xmlns:wsu="http://schemas.xmlsoap.org/ws/2002/07/utility"
  xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/07/secext">
  <soap:Header>
    <wsse:Security>
      <wsse:UsernameToken>
        <wsse:Username>USERNAME</wsse:Username>
        <wsse:Password Type="wsse:PasswordDigest">PASSWORDDIGEST</wsse:Password>
        <wsse:Nonce>NONCE</wsse:Nonce>
        <wsu:Created>2003-09-08T05:52:36Z</wsu:Created>
        <wsu:Expires>2003-09-08T05:55:36Z</wsu:Expires>
      </wsse:UsernameToken>
    </wsse:Security>
  </soap:Header>
  <soap:Body>
    <GetVersion xmlns="http://www.microsoft.com">
    </GetVersion>
  </soap:Body>
</soap:Envelope>

…and you get this…

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8

<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soap:Body>
    <GetVersionResponse xmlns="http://www.microsoft.com">
      <GetVersionResult>Microsoft.Com Platform Services 1.0 Beta</GetVersionResult>
    </GetVersionResponse>
  </soap:Body>
</soap:Envelope>

In the REST version, you send this:

GET /msweb/Version HTTP/1.0
X-WSSE: WSSE Username="USERNAME", PasswordDigest="PASSWORDDIGEST", Nonce="NONCE", Created="2003-09-08T05:52:36Z", Expires="2003-09-08T05:55:36Z"

…and you get this…

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8

<GetVersionResult xmlns="http://www.microsoft.com">Microsoft.Com Platform Services 1.0 Beta</GetVersionResult>

I know, I know, you couldn’t care less about the wire formats, because your fancy WSDL-enabled IDE hides all of that nastiness from you and lets you call auto-generated wrapper classes that Just Work, and gives you Intellisense popups too. Well, just for you, we created a WSDL file for our REST-based service, which has all the features and benefits of the WSDL for the original SOAP-based service. (Few people know that the WSDL specification supports HTTP GET bindings. Even Visual Studio .NET supports them. Intellisense, here we come.) Joe and I tested our WSDL file in both VS.NET 2002 and 2003. Here’s some C# code that Joe wrote that calls the REST web service via a C# wrapper class that wsdl.exe auto-generated from our WSDL file. I was there at the time, and I can attest to the fact that it Just Worked. It was a beauty to behold. The tools may save us after all… At the very least, Joel can have his Intellisense popups (which are enabled by WSDL, not SOAP), and the rest of us can have a cleaner wire format and less interop pain. The bits on the wire matter, Joel. The bits on the wire always matter.

I am indebted to Paul Prescod for his work on the Google API, which taught me how to write a WSDL file with HTTP bindings that worked in Visual Studio .NET.

§

Twenty four comments here (latest comments)

1. Nice work Mark (and Joe). Really nice work.

   — Walt 

2. I’m missing something here. I’ve clicked on the URL, and compiled and run the SampleCSClient. In neither case am I seeing any X-WSSE headers nor challenges. I’m particularly interested in seeing how the SampleCSClient would change to support authentication.

   Here is a sample wire trace:

   GET /msweb/Version HTTP/1.1
   Host: diveintomark.org
   User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3) Gecko/20030312
   Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1
   Accept-Language: en-us,en;q=0.5
   Accept-Encoding: gzip,deflate,compress;q=0.9
   Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   Keep-Alive: 300
   Connection: keep-alive

   HTTP/1.1 200 OK
   Date: Mon, 08 Sep 2003 08:17:25 GMT
   Server: Apache/1.3.27 (Unix) (Red-Hat/Linux) PHP/4.1.2 mod_gzip/1.3.26.1a DAV/1.0.3 mod_ssl/2.8.12 OpenSSL/0.9.6b mod_webkit/0.5
   Etag: “f93d0613cd8ecbdfc35e90ff9bad07c849f0f420″
   Vary: Accept-Encoding
   Content-Length: 110
   Keep-Alive: timeout=5, max=5
   Connection: Keep-Alive
   Content-Type: text/xml;charset=utf-8

   <GetVersionResult xmlns=”http://www.microsoft.com“>Microsoft.Com Platform Services 1.0 Beta</GetVersionResult>

   — Sam Ruby 

3. I suspect that if I understood more than 1/10 of this post I would think “How cool!”

   — xian 

4. What happens to large, comples data structures that need to be shuttled back and forth?

   I understand that stuff you receive will be in the output body, but stuff you send? URL length is limited…

   (I’m not a SOAP advocate BTW)

   — Duncan Wilcox 

5. A small suggestion: I’d prefer that the HTTP header equivalent of the wsse:UsernameToken be “X-WSSE UsernameToken” instead of “X-WSSE WSSE”. In particular, this would leave open the option of later supporting BinarySecurityToken for things like Kerberos.

   http://www-106.ibm.com/developerworks/webservices/library/ws-secure/#minorhead4.2

   — Sam Ruby 

6. Bravo!

   The next question is, why do they require you to authenticate yourself in order to find out what the top downloads are…? :-)

   — Damian Cugley 

7. Sam: good point about the auth. The reason you’re not seeing it is because the gateway server is defaulting the authentication for the purposes of this demonstration. In the msproxy.py script, comment out the getDefaultAuthParams call and uncomment out the failure(401) line below it. I’ll make a note of this in the text.

   Also a good point about the auth header name.

   — Mark 

8. Duncan: it’s possible that future revisions of the API would contain functions that took so many parameters that you would need to use POST and encode the parameters in the body (think HTML form submission). The HTTP bindings for WSDL have anticipated this need and support POST for requests as well as GET.

   As for complex data structures going in, the WSDL includes a schema that can define arbitrary data structures for input as well as output. I’m not sure what it would end up looking like over the wire (I’ve just never done it). However, none of the current functions were anywhere close to taking complex data types; by defining guids for each download and passing those around instead, they seem intent on *avoiding* such a scenario.

   — Mark 

9. So… that’s why the SampleCSClient “just worked”.

   Any possibility of producing a version of this client which actually makes use of the authentication headers you are proposing?

   — Sam Ruby 

10. As to the question of how to support the custom auth in C#, I would also like to know this. My C# experience is pretty much limited to looking over Joe’s shoulder (although I will have more exposure in the coming months). In the meantime, if some enterprising young .NET guru wanted to take a look at the code and improve it to send custom authentication information in an HTTP header, and in the right format, I would be most grateful.

    Note that in MS’s original version, authentication was handled by a library in the Microsoft Web Services Enhancement Service Pack, and as I recall, I had to download the latest version of that before their sample code would work. It may be possible to reuse that code and simply put the result in the HTTP header instead; the relevant library is Microsoft.Web.Services.Security::UsernameToken.

    — Mark 

11. Nice work on the WSDL! Have you tested which programming environments it works with? Java’s Apache Axis 1.1 doesn’t seem to generate any useful code from your WSDL. Perl’s SOAP::Lite 0.55 doesn’t seem to find any methods in the WSDL either.

    WSDL is one of those squishy specs where stuff may be ‘correct’ but it doesn’t work with any of the tools. I suspect the issue here is since no one is using the GET binding, it just doesn’t work in many places. Frustrating situation.

    — Nelson 

12. I blogged a bit at http://www.nelson.monkey.org/~nelson/weblog/tech/wsdlGetBinding.html

    — Nelson 

13. Well, so much for “the tools will save us”. Although I’m not sure why you’re trying to use a SOAP tool to access a REST web service.

    Regardless, luckily for the Perl users of the world, the simpler the wire format and protocol, the easier it is to code to ourselves while we wait for the tools to catch up.

    — Mark 

14. Mark,
    I haven’t looked at the code but if you mean the nonce & password digest when you say custom authentication information then you should be able to figure out how to do this from my code that converts the Microsoft.com Web Service to an RSS feed at http://www.kuro5hin.org/story/2003/9/4/122544/7625
    Works without using the Web Services Enhancements, just vanilla .NET Framework.

    — Dare Obasanjo 

15. use sarcasm;

    Hey, what’s wrong with Perl? I’ll tell ya. Perl users are ten times smarter than any Python user can ever be. We also cook eggs faster. :)

    — Jesper 

16. mad. props.

    — sleeper 

17. The rationale for using a SOAP header for authentication information is that behind the service might be non-HTTP transports.

    — Rich Salz 

18. Rich,
    That rationalization is just that, a rationalization, since the only SOAP binding today is HTTP. Also, Mark and I clearly have shown how to work within HTTP to add a new authentication mechanism if one is needed.

    No matter what protocol SOAP is run over the first thing you are going to do is throw away the native authentication. Want to run SOAP over Jabber, sure, first throw out the authentication work they’ve already done on Jabber. Ditto for HTTP. Ditto for SMTP. Ditto for AIM. This is what I call “the arrogance of SOAP”.

    — Joe 

19. No, it’s a justification. Just because W3C has only issued one binding, doesn’t mean that SOAP can’t run over MQ, JMS, MSMQ/COM+, etc. It does, and putting auth information into SOAP as opposed to its transport gives end-to-end identity, integrity, etc.

    — Rich Salz 

20. Radovan Janecek: Nothing Impersonal (trackback)

21. Cool. This is exactly how web services should be exposed – REST frontend and interfaces on the backend side…

    — Radovan Janecek 

22. Rich,
    Thanks for confirming that you intend to throw out all the hardwork of every protocol SOAP comes in contact with.

    “putting auth information into SOAP as opposed to its transport gives end-to-end identity, integrity, etc.”

    So for every protocol SOAP is used over you propose to re-invent the wheel regardless of the quality of the native identity, integrity and authentication mechanism?

    In other words, SOAP is the Swing of Web Services.

    — Joe 

23. “Attacking or trying to tear down other peoples work is non-productive and plainly not my style.”

    - http://wellformedweb.org/news/1

    — Sam Ruby 

24. Sam, the issue at hand is SOAP and its document-level overrides of native transport mechanisms. I think the analogy to Swing is quite apt. Have you ever seen Swing on Mac OS X? It’s completely integrated with the underlying operating system’s graphical interface. A Swing application on OS X looks like a native OS X (Cocoa) application, because Apple worked with Sun to produce a custom JVM that uses the underlying menus, buttons, scroll bars, windows, etc. instead of reinventing the wheel. So developers get the benefits of cross-platform coding, and end users get the benefits of their own platform.

    Would a similar thing be possible here? SOAP headers that just specify that such-and-such an algorithm should be used to encode certain information *in the underlying transport’s framework*. Your fancy SOAP libraries can hide the details, of course.

    Just an off-the-cuff idea. Maybe I’m taking the analogy too far. Regardless, Joe’s point stands unrefuted: SOAP is rude, and if by a sheer miracle of fate you actually *do* ever get to reuse the same SOAP API over a different protocol, you’re going to have all these arguments over again with the next community you step on.

    — Mark 

Respond privately

I am no longer accepting public comments on this post, but you can use this form to contact me privately. (Your message will not be published.)

§