Archives
-
Azure Mobile Services: revealing the internals
I think this is an interesting blog post if you want to know how azure Mobile Services is working. Don’t forget to check the link at the end of this post to see all source code of Azure Mobile Services:-)
The Azure Mobile Services documentation is … suboptimal. It is unclear in which context our server side scripts are running, which node.js modules are available, and which node.js modules are already loaded. The best way to get an idea on what is going on is by looking at the source code that is running inside Azure Mobile Services. In this post I will show some of the node.js scripts running Azure Mobile Services. These scripts are the scripts running today, but they may change as we speak. At least the scripts will give us some ideas what is happening.
I will try to comment on some things that we see, but I’m absolutely no node.js expert, I just started to dive into this technology.
Azure mobile services in built on top of node.js which is running in IIS8 using iisnode. To support the web features like the API’s Azure Mobile Services uses the Express web application framework for node.
When you look at the getting started guide of Express you see that you need to “… Create a file named app.js or server.js”. If you look at the used Web.config file this is exactly the starting point that is on Azure Mobile Services as defined in the handlers section: <handlers> <add name="iisnode" path="app.js" verb="*" modules="iisnode" /> </handlers>
Source-code of Web.config:
<?xml version="1.0"?> <configuration> <system.web> <customErrors mode="Off" /> </system.web> <system.webServer> <httpErrors errorMode="Detailed" /> <iisnode devErrorsEnabled="false" loggingEnabled="false" /> <handlers> <add name="iisnode" path="app.js" verb="*" modules="iisnode" /> </handlers> <rewrite> <rules> <rule name="favicon"> <match url="favicon.ico" /> <action type="CustomResponse" statusCode="404" /> </rule> <!-- Serve a robots.txt file which disallows everything --> <rule name="robots" stopProcessing="true"> <match url="^robots.txt" /> <action type="Rewrite" url="static/robots.txt" /> </rule> <rule name="landingpage" stopProcessing="true"> <match url="^$" /> <action type="Rewrite" url="static/default.htm" /> </rule> <!-- This rule allows IIS to serve the /static/client directory natively --> <rule name="client" stopProcessing="true"> <match url="^client/(.*)" /> <action type="Rewrite" url="static/client/{R:1}" /> </rule> <!-- This rule routes everything to app.js, except for direct requests to app.js, as in /app.js/debug node debugging. --> <rule name="app"> <match url="/*" /> <action type="Rewrite" url="app.js" /> </rule> </rules> </rewrite> </system.webServer> <location path="runtime"> <system.web> <authorization> <deny users="*" /> </authorization> </system.web> </location> <!-- The below settings are for local testing only. The web config transforms remove all of these settings when the deploy package is built, since placeholder setting values arent needed in Antares --> <appSettings> <!-- This setting must remain, so the package process can set the version --> <add key="RuntimeVersion" value="Zumo.Main.0.1.6.3017.Runtime" /> </appSettings> </configuration>
What more do we see in the Web.config file:
- There is a static folder that is used for serving static files
- /robots.txt is mapped to static/robots.txt
- the root (/) of the Mobile Services Web site is mapped to static/default.htm
- the client JavaScript libraries are mapped to /static/client/*
- /client/MobileServices.Web-1.0.0.js
<li>/client/MobileServices.Web-1.0.0.min.js</li>
Ok, so everything kicks off in a node.js file call app.js. So lets get started here...
Source-code of app.js:
// ---------------------------------------------------------------------------- // Copyright (c) Microsoft Corporation. All rights reserved. // ---------------------------------------------------------------------------- // // The Zumo runtime. Creates an instance of a Zumo server with options determined // by env variables and starts listening on the port designated by the PORT env // variable. require('./runtime/server.js').createServer(process.env).listen(process.env.PORT);
From this simple app.js script a module called server.js is loaded, and in this module the createServer() function is called with a lot of environment settings.
In general these environment settings look like: (I anonymized some values)
{ "APP_POOL_CONFIG": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\Config\\applicationhost.config", "APP_POOL_ID": "<NAME-OF-AZURE-MOBILE-SERVICES-SITE>", "PROCESSOR_ARCHITEW6432": "AMD64", "TMP": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\Temp", "TEMP": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\Temp", "ApplicationName": "<NAME-OF-AZURE-MOBILE-SERVICES-SITE>", "APPSETTING_ApplicationName": "<NAME-OF-AZURE-MOBILE-SERVICES-SITE>", "UserConnectionString": "<MY-DATABASE-CONNECTIONSTRING>", "APPSETTING_UserConnectionString": "<MY-DATABASE-CONNECTIONSTRING>", "LogServiceURL": "https://<NAME-OF-LOGSERVICE-SERVER>.cloudapp.net/", "APPSETTING_LogServiceURL": "https://<NAME-OF-LOGSERVICE-SERVER>.cloudapp.net/", "LogServiceToken": "<MY-LOGSERVICE-TOKEN>", "APPSETTING_LogServiceToken": "<MY-LOGSERVICE-TOKEN>", "ApnsCertificateMode": "None", "APPSETTING_ApnsCertificateMode": "None", "ApnsCertificatePassword": "", "APPSETTING_ApnsCertificatePassword": "", "ApplicationKey": "<MY-APPLICATIONKEY>", "APPSETTING_ApplicationKey": "<MY-APPLICATIONKEY>", "ApplicationMasterKey": "<MY-APPLICATIONMASTERKEY>", "APPSETTING_ApplicationMasterKey": "<MY-APPLICATIONMASTERKEY>", "LogLevel": "Error", "APPSETTING_LogLevel": "Error", "DynamicSchemaEnabled": "True", "APPSETTING_DynamicSchemaEnabled": "True", "PreviewFeatures": "[]", "APPSETTING_PreviewFeatures": "[]", "ApplicationSystemKey": "<MY-APPLICATIONSYSTEMKEY>", "APPSETTING_ApplicationSystemKey": "<MY-APPLICATIONSYSTEMKEY>", "ScmType": "None", "APPSETTING_ScmType": "None", "APPDATA": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\AppData", "LOCALAPPDATA": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\LocalAppData", "PROGRAMDATA": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\ProgramData", "ALLUSERSPROFILE": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\ProgramData", "USERPROFILE": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\UserProfile", "HOME": "C:\\DWASFiles\\Sites\\<NAME-OF-AZURE-MOBILE-SERVICES-SITE>\\VirtualDirectory0", "windows_tracing_flags": "", "windows_tracing_logfile": "", "Path": "D:\\Windows\\system32;D:\\Windows;D:\\Windows\\System32\\Wbem;D:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;D:\\Users\\OnStartAdmin\\AppData\\Roaming\\npm;D:\\Program Files (x86)\\nodejs\\;D:\\Program Files (x86)\\Mercurial\\;D:\\Program Files (x86)\\Microsoft ASP.NET\\ASP.NET Web Pages\\v1.0\\;D:\\Program Files (x86)\\PHP\\v5.3;", "CommonProgramFiles": "D:\\Program Files (x86)\\Common Files", "CommonProgramFiles(x86)": "D:\\Program Files (x86)\\Common Files", "CommonProgramW6432": "D:\\Program Files\\Common Files", "COMPUTERNAME": "<MY-COMPUTERNAME>", "ComSpec": "D:\\Windows\\system32\\cmd.exe", "FP_NO_HOST_CHECK": "NO", "NUMBER_OF_PROCESSORS": "8", "OS": "Windows_NT", "PATHEXT": ".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC", "PROCESSOR_ARCHITECTURE": "x86", "PROCESSOR_IDENTIFIER": "AMD64 Family 16 Model 4 Stepping 2, AuthenticAMD", "PROCESSOR_LEVEL": "16", "PROCESSOR_REVISION": "0402", "ProgramFiles": "D:\\Program Files (x86)", "ProgramFiles(x86)": "D:\\Program Files (x86)", "ProgramW6432": "D:\\Program Files", "PSModulePath": "D:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules\\", "PUBLIC": "D:\\Users\\Public", "SystemDrive": "D:", "SystemRoot": "D:\\Windows", "USERDOMAIN": "WORKGROUP", "USERNAME": "<MY-USERNAME>", "windir": "D:\\Windows", "PORT": "\\\\.\\pipe\\b4b47d62-dcfe-426b-a239-6a4755a06f29", "IISNODE_VERSION": "0.2.6", "RuntimeVersion": "Zumo.Main.0.1.6.3221.Runtime" }
Hey, Azure runs on 8-processors 64-bit AMD processors!
But lets not get distracted, the real thing gets going in server.js…
Source-code of ./runtime/server.js:
// ---------------------------------------------------------------------------- // Copyright (c) Microsoft Corporation. All rights reserved. // ---------------------------------------------------------------------------- // // Defines the Zumo runtime HTTP server, which delegates to the request handler. var path = require('path'), RequestHandler = require('./request/requesthandler'), Request = require('./request/request'), Logger = require('./logger'), StatusCodes = require('./statuscodes').StatusCodes, core = require('./core'), tripwire = require('tripwire'), Metrics = require('./metrics'), ScriptManager = require('./script/scriptmanager'), resource = require('./resources'), _ = require('underscore'), _str = require('underscore.string'), express = require('express'); _.mixin(_str.exports()); var logSource = 'Server'; exports.createServer = createServer; function createServer(options) { Logger.initialize(options.logServiceURL, options.logServiceToken); var configPath = path.join(__dirname, '..', options.dataDirectory || './App_Data', 'config'), globalLogger = new Logger(LogLevel[options.logLevel]), metrics = new Metrics(globalLogger, parseInt(options.metricsTimeout || 300000, 10)), // Five minutes default maxRequestBodySize = (options.maxRequestBodySizeKB || 1024) * 1024, authenticationCredentials = getAuthenticationCredentials(options), crossDomainWhitelist = getCrossDomainWhitelist(globalLogger); var scriptManager = new ScriptManager(configPath, options.userConnectionString, options.applicationName, core.parseBoolean(options.dynamicSchemaEnabled), authenticationCredentials, options.ApnsCertificatePassword, options.ApnsCertificateMode, globalLogger, metrics); var requestHandler = new RequestHandler(configPath, options.applicationMasterKey, options.applicationSystemKey, options.applicationName, authenticationCredentials, crossDomainWhitelist, options.userConnectionString, options.applicationKey, core.parseBoolean(options.dynamicSchemaEnabled), options.runtimeVersion, options.requestTimeout, scriptManager, globalLogger, metrics, options.logLevel, options.logServiceURL, options.logServiceToken, maxRequestBodySize); var app = express(); var server; if (options.pfx) { server = require('https').createServer({ pfx: options.pfx, passphrase: options.passphrase }, app); } else { server = require('http').createServer(app); } Logger.writer.on('error', function () { // ignore failures. we need this handler here to prevent errors // from bubbling up to the global exception handler, which would // cause the process to be killed }); registerUncaughtExceptionListener(server, options, globalLogger, metrics); // override the listen function so the server only starts listening // once all async initialization is complete var originalListen = server.listen; server.listen = function () { var listenArgs = arguments; requestHandler.initialize(app, function () { scriptManager.runStartupScript(3600000); originalListen.apply(server, listenArgs); }); }; return server; } function registerUncaughtExceptionListener(server, options, logger, metrics) { var tripwireContext = {}; var tripwireKeepalive = parseInt(options.tripwireKeepalive, 10) || 1000; var processShutdownTimeout = parseInt(options.processShutdownTimeout, 10) || 2000; // 99.9% of the time async errors will end up here and we assume all async errors belong to user code var onUncaughtException = function (e) { process.removeAllListeners('uncaughtException'); var exitCode; var isTripWireError = false; if (tripwireContext === tripwire.getContext()) { e = new Error(_.sprintf(resource.tripwireError, tripwireKeepalive)); exitCode = 2; isTripWireError = true; } else { e = e || new Error('The application generated an unspecified exception.'); exitCode = 1; } logGlobalException(e, isTripWireError, logger); // flush any pending global log operations Logger.flush(); // Wait a short period of time to allow any other logger instances a chance // to flush themselves (based on their flush timeouts). setTimeout(function () { process.exit(exitCode); }, processShutdownTimeout); }; process.on('uncaughtException', onUncaughtException); function resetTripwire() { tripwire.resetTripwire(tripwireKeepalive * 2, tripwireContext); } resetTripwire(); var tripwireInterval = setInterval(resetTripwire, tripwireKeepalive); server.on('close', function () { tripwire.clearTripwire(); clearInterval(tripwireInterval); process.removeListener('uncaughtException', onUncaughtException); }); } function logGlobalException(e, isTripWireError, logger) { if (!isTripWireError && core.isRuntimeError(e)) { logger.error(logSource, e); } else { var userScriptSource = core.parseUserScriptError(e); var stackPrefix = userScriptSource ? '' : 'An unhandled exception occurred. '; var stack = e.stack ? stackPrefix + e.stack : ''; var errMsg = stack || e.message || e.toString(); logger.logUser(userScriptSource, 'error', errMsg); } } function getAuthenticationCredentials(options) { var result = { microsoftaccount: { clientId: process.env.WLClientId, clientSecret: process.env.WLClientSecret || options.wLClientSecret, packageSid: process.env.WLPackageSid || options.wLPackageSid }, facebook: { appId: process.env.AUTH_FACEBOOK_APPID, appSecret: process.env.AUTH_FACEBOOK_SECRET }, twitter: { consumerKey: process.env.AUTH_TWITTER_APPID, consumerSecret: process.env.AUTH_TWITTER_SECRET }, google: { clientId: process.env.AUTH_GOOGLE_APPID, clientSecret: process.env.AUTH_GOOGLE_SECRET, gcmApiKey: process.env.GCM_API_KEY } }; result.microsoftaccount.enabled = isProviderEnabled(result.microsoftaccount.clientId, result.microsoftaccount.clientSecret); result.facebook.enabled = isProviderEnabled(result.facebook.appId, result.facebook.appSecret); result.twitter.enabled = isProviderEnabled(result.twitter.consumerKey, result.twitter.consumerSecret); result.google.enabled = isProviderEnabled(result.google.clientId, result.google.clientSecret); return result; // a certain auth credential provider is enabled if all required fields are non empty strings function isProviderEnabled(id, secret) { if (typeof (id) !== "string" || id.length === 0) { return false; } if (typeof (secret) !== "string" || secret.length === 0) { return false; } return true; } } function getCrossDomainWhitelist(logger) { var serializedValue = process.env.CrossDomainWhitelist; if (serializedValue) { try { return JSON.parse(serializedValue); } catch (ex) { ex.message += "; Attempted JSON value: " + serializedValue; logger.error(logSource, ex); } } return null; }
As you can see in the call from App.js to the function createServer(options), the options are actually Process.Env, the environment settings. So all the configuration settings you can do in the Azure Mobile Services UI like the DynamicSchemaEnabled are just persisted to environment variables. If a value is not set yet (like in the above case the authentication credentials), the enviroment variable does not exist yet.
In the above code you see that the options parameter (containing a hashtable with all environment variable settings) and Process.Env are used mixed. A good example is the function getAuthenticationCredentials(options), the options (Process.Env) are passed in as a parameter, but it still reads the settings directly from Process.Env.
If you look at the code above, a lot of things are initialized based on the enviroment variables. If environment variables are changed through the Azure Mobile Services Web UI, the process must be restarted to pick-up the new environment settings an reinitialize the application. I did not investigate this further yet.
If you look at the above code the following roughly happening:
- A lot of node.js modules are loaded
- The server is instantiated
- Log service is initialized
- Global variables like authentication configuration and the cross domain white list (CORS) are set
- The scriptmanager (wires the api, table and scheduler scripts)
- The requesthandler is initialized, this handles the request that come in and uses the scriptmanager for redirecting to api, table or scheduler scripts
- The Express() web server system is started
- Top-level Exception handling is configured
- When all (async) initialization is completed the Express server starts listening to requests and we are in business
From there on a lot of stuff is happening. Too much to describe in this blog post, and too much for me to understand in detail. I think it is better to have a look for yourself, check out https://documented.azure-mobile.net/api/dir. Let me know what you think of this:-)