Azure: New Marketplace, Network Improvements, New Batch Service, Automation Service, more

Tuesday, October 28, 2014

Today we released a major set of updates to Microsoft Azure. Today’s updates include:

Marketplace: Announcing Azure Marketplace and partnerships with key technology partners
Networking: Network Security Groups, Multi-NIC, Forced Tunneling, Source IP Affinity, and much more
Batch Computing: Public Preview of the new Azure Batch Computing Service
Automation: General Availability of the Azure Automation Service
Anti-malware: General Availability of Microsoft Anti-malware for Virtual Machines and Cloud Services
Virtual Machines: General Availability of many more VM extensions – PowerShell DSC, Octopus, VS Release Management

All of these improvements are now available to use immediately (note that some features are still in preview). Below are more details about them:

Marketplace: Announcing Azure Marketplace and partnerships with key technology partners

Last week, at our Cloud Day event in San Francisco, I announced a new Azure Marketplace that helps to better connect Azure customers with partners, ISVs and startups. With just a couple of clicks, you can now quickly discover, purchase, and deploy any number of solutions directly into Azure.

Exploring the Marketplace

You can explore the Azure Marketplace by clicking the Marketplace title that is pinned by default to the home-screen of the Azure Preview Portal:

Clicking the Marketplace tile will enable you to explore a large selection of applications, VM images, and services that you can provision into your Azure subscription:

Using the marketplace provides a super easy way to take advantage of a rich ecosystem of applications and services integrated to run great with Azure. Today’s marketplace release includes multi-VM templates to run Hadoop clusters powered by Cloudera or Hortenworks, Linux VMs powered by Unbuntu, CoreOS, Suse, CentOS, Microsoft SharePoint Server Farms, Cassandra Clusters powered by DataStax, and a wide range of security virtual appliances.

You can click any of the items in the gallery to learn more about them and optionally deploy them. Doing so will walk you though a simple to follow creation wizard that enables you to optionally configure how/where they will run, as well as display any additional pricing required for the apps/services/VM images that you select.

For example, below is all it takes to stand-up an 8-node DataStax Enterprise cluster:

Solutions you purchase through the Marketplace will be automatically billed to your Azure subscription (avoiding the need for you to setup a separate payment method). Virtual Machine images will support the ability to bring your own license or rent the image license by the hour (which is ideal for proof of concept solutions or cases where you need the solution for only a short period of time). Both Azure Direct customers as well as customers who pay using an Enterprise Agreement can take advantage of the Azure Marketplace starting today.

You can learn more about the Azure Marketplace as well as browse the items within it here.

Networking: Lots and lots of New Features and Improvements

This week’s Azure update includes a ton of new capabilities to the Azure networking stack. You can use these new networking capabilities immediately in the North Europe region, and they will be supported worldwide in all regions in November 2014. The new network capabilities include:

Network Security Groups

You can now create Network Security groups to define access control rules for inbound and outbound traffic to a Virtual machine or a group of virtual machines in a subnet. The security groups and the rules can be managed and updated independent of the life cycle of the VM.

Multi-NIC Support

You can now create and manage multiple virtual network interfaces (NICs) on a VM. Multi-NIC support is a fundamental requirement for a majority of network virtual appliances that can be deployed in Azure. Having this support now enabled within Azure will enable even richer network virtual appliances to be used.

Forced Tunneling

You can now redirect or “force” all Internet-bound traffic that originates in a cloud application back through an on-premises network via a Site-to-Site VPN tunnel for inspection and auditing. This is a critical security capability for enterprise grade applications.

ExpressRoute Enhancements

You can now share a single ExpressRoute connection across multiple Azure subscriptions. Additionally, a single Virtual Network in Azure can now be linked to more than one ExpressRoute circuit, thereby enabling much richer backup and disaster recovery scenarios.

New VPN Gateway Sizes

To cater to the growing hybrid connectivity throughput needs and the number of cross premise sites, we are announcing the availability of a higher performance Azure VPN gateway. This will enable a faster ExpressRoute and Site-to-Site VPN gateways with more tunnels.

Operations and audit logs for VNet Gateways and ExpressRoute

You can now view operations logs for Virtual Network Gateways and ExpressRoute circuits. The Azure portal will now show operations logs and information on all API calls you make and important infrastructure changes made such as scheduled updates to gateways.

Advanced Virtual Network Gateway policies

We now enable the ability for you to control encryption for the tunnel between Virtual Networks. You now have a choice between 3DES, AES128, AES256 and Null encryption, and you can also enable Perfect Forward Secrecy (PFS) for IPsec/IKE gateways.

Source IP Affinity

The Azure Load Balancer now supports a new distribution mode called Source IP Affinity (also known as session affinity or client IP affinity). You can now load balance traffic based on a 2-tuple (Source-IP, Destination-IP) or 3-tuple (Source-IP, Destination-IP and Protocol) distribution modes.

Nested policies for Traffic Manager

You can now create nested policies for traffic management. This allows tremendous flexibility in creating powerful load-balancing and failover schemes to support the needs of larger, more complex deployments.

Portal Support for Managing Internal Load Balancer, Reserved and Instance IP addresses for Virtual Machines

It is now possible to use the Azure Preview Portal to manage creating and setting up internal load balancers, as well as reserved and instance IP addresses for virtual machines.

Automation: General Availability of Azure Automation Service

I am excited to announce the General Availability of the Azure Automation service. Azure Automation enables the creation, deployment, monitoring, and maintenance of resources in an Azure environment using a highly scalable and reliable workflow engine. The service can be used to orchestrate time-consuming and frequently repeated operational tasks across Azure and third-party systems while decreasing operating expenses.

Azure Automation allows you to build runbooks (PowerShell Workflows) to describe your administration processes, provides a secure global assets store so you don’t need to hardcode sensitive information within your runbooks, and offers scheduling so that runbooks can be triggered automatically.

Runbooks can automate a wide range of scenarios – from simple day to day manual tasks to complex processes that span multiple Azure services and 3rd party systems. Because Automation is built on PowerShell, you can take advantage of the many existing PowerShell modules, or author your own to integrate with third party systems.

Creating and Editing Runbooks

You can create a runbook from scratch, or start by importing an existing template in the runbook gallery:

Editing experience for runbooks can also be performed directly in the administration portal:

Pricing

Available as a pay-as-you-go service, Automation is billed based on the number of job run time minutes used in a given Azure subscription. 500 minutes of free job runtime credits are also included each month for Azure customers to use at no charge.

Learn More

To learn more about Azure Automation, check out the following resources:

Batch Service: Preview of Azure Batch - new job scheduling service for parallel and HPC apps

I’m excited to announce the public preview of our new Azure Batch Service. This new platform service provides “job scheduling as a service” with auto-scaling of compute resources, making it easy to run large-scale parallel and high performance computing (HPC) work in Azure. You submit jobs, we start the VMs, run your tasks, handle any failures, and then shut things down as work completes.

Azure Batch is the job scheduling engine that we use internally to manage encoding for Azure Media Services, and for testing Azure itself. With this preview, we are excited to expand our SDK with a new application framework from GreenButton, a company Microsoft acquired earlier in the year. The Azure Batch SDK makes it easy to cloud-enable parallel, cluster, and HPC applications by describing jobs with the required resources, data, and one or more compute tasks.

Azure Batch can be used to run large volumes of similar tasks or applications in parallel, programmatically. A command line program or script takes a set of data files as input, processes the data in a series of tasks, and produces a set of output files. Examples of batch workloads that customers are running today in Azure include calculating risk for banks and insurance companies, designing new consumer and industrial products, sequencing genes and developing new drugs, searching for new energy sources, rendering 3D animations, and transcoding video.

Azure Batch makes it easy for these customers to use hundreds, thousands, tens of thousands of cores, or more on demand. With job scheduling as a service, Azure developers can focus on using batch computing in their applications and delivering services without needing to build and manage a work queue, scaling resources up and down efficiently, dispatching tasks, and handling failures.

The scale of Azure helps batch computing customers get their work done faster, experiment with different designs, run larger and more precise models, and test a large number of different scenarios without having to invest in and maintain large clusters.

Learn more about Azure Batch and start using it for your applications today.

Virtual Machines: General Availability of Microsoft Anti-Malware for VMs and Cloud Services

I’m excited to announce that the Microsoft Anti-malware security extension for Virtual Machines and Cloud Services is now generally available. We are releasing it as a free capability that you can use at no additional charge.

The Microsoft Anti-malware security extension can be used to help identify and remove viruses, spyware or other malicious software. It provides real-time protection from the latest threats and also supports on-demand scheduled scanning. Enabling it is a good security best practice for applications hosted either on-premises or in the cloud.

Enabling the Anti-Malware Extension

You can select and configure the Microsoft Antimalware security extension for virtual machines using the Azure preview portal, Visual Studio or API’s/PowerShell. Antimalware events are then logged to the customer configured Azure Storage account via Azure Diagnostics and can be piped to HDInsight or a SIEM tool for further analysis. More information is available in the Microsoft Antimalware Whitepaper.

To enable antimalware feature on existing virtual machine, select the EXTENSIONS tile on a Virtual Machine in the Azure Preview Portal, then click ADD in the command bar and select the Microsoft Antimalware extension. Then, click CREATE and customize any settings:

Virtual Machines: General Availability of even more VM Extensions

In addition to enabling the Microsoft Anti-Malware extension for Virtual Machines, today’s release also includes support for a whole bunch more new VM extensions that you can enable within your Virtual Machines. These extensions can be added and configured using the same EXTENSIONS tile on Virtual Machine resources within the Azure Preview Portal (the same screen-shot as in the Anti-malware section above).

The new extensions enabled today include:

PowerShell Desired State Configuration

The PowerShell Desired State Configuration Extension can be used to deploy and configure Azure VMs using Desired State Configuration (DSC) technology. DSC enables you to declaratively specify how you want your software environment to be configured. DSC configuration can also be automated using the Azure PowerShell SDK, and you can push configurations to any Azure VM and have them enacted automatically. For more details, please see this desired state configuration blog post.

Octopus

Octopus simplifies the deployment of ASP.NET web applications, Windows Services and other applications by automatically configuring IIS, installing services and making configuration changes. Octopus integration of Azure was one of the top requested features on Azure UserVoice and with this integration we will simplify the deployment and configuration of octopus on the VM.

Visual Studio Release Management

Release Management for Visual Studio is a continuous delivery solution that automates the release process through all of your environments from TFS through to production. Visual Studio Release Management is integrated with TFS and you can configure multi-stage release pipelines to automatically deploy and validate your applications on multiple environments. With the new Visual Studio Release Management extension, VMs can be preconfigured with the necessary components for required for Release Management to operate.

Summary

Today’s Microsoft Azure release enables a ton of great new scenarios, and makes building applications hosted in the cloud even easier.

If you don’t already have a Azure account, you can sign-up for a free trial and start using all of the above features today. Then visit the Microsoft Azure Developer Center to learn more about how to build apps with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu

Enterprises in Southern California and the South West regions have enjoyed the DataStax / Azure collaboration. 949.861.1069 | scott.herson@datastax.com

Scott Herson - Tuesday, October 28, 2014 4:05:27 PM

Any information about how to add multiple NICs to VMs? Also, by VM do you also mean Cloud Service VMs?

Zoltán Lehóczky - Tuesday, October 28, 2014 8:56:26 PM

Great! Now I need static MAC and console access to a VM and I am ready to move to the cloud.

AzureLover - Tuesday, October 28, 2014 9:30:19 PM

In response to the question from Zoltán - you can find instructions on adding multiple NICs on MSDN here: http://msdn.microsoft.com/en-us/library/dn848315.aspx (Note you'll need the latest PowerShell Cmdlets and be running the commands in North Europe for now).

Simon Waight - Tuesday, October 28, 2014 10:57:34 PM

where are the details for the sticky session load balancing configuration

CJR - Wednesday, October 29, 2014 12:16:38 AM

found the LB details http://msdn.microsoft.com/en-us/library/dn655055.aspx

cjr - Wednesday, October 29, 2014 12:20:05 AM

I have been waiting for this feature for a long time!

Was I the only one that was really hoping (presuming!) it would have been possible to route traffic across the interfaces though...

AZL - Wednesday, October 29, 2014 12:24:21 AM

Very informtive post.I like it.

jackshron - Wednesday, October 29, 2014 7:18:57 AM

In response to AZL, although we do not support routing through the multiple interfaces at the moment, we have received the feedback on the need for this feature. We hope to enable it in the near future.

Yousef Khalidi - Wednesday, October 29, 2014 9:23:38 AM

@Simon Waight thank you! It seems that the multi-NIC support is very limited. Not just it's available only in one region but you can't assign public VIPs to more than one NIC and finally you can only use multiple NICs from Large VM size up.

Zoltán Lehóczky - Wednesday, October 29, 2014 10:51:13 AM

@Zoltan, the new features are currently being rolled out to all regions. This link: http://azure.microsoft.com/en-us/updates/?service=virtual-network will carry an update when the rollout is complete. Also, we are working on enhancing the feature set for Multiple NICs. Thanks.

Narayan_Msft - Wednesday, October 29, 2014 11:03:46 AM

Great information.

partha - Thursday, October 30, 2014 11:27:39 AM

How would I add the anti-malware protection to a cloud service?

Michael - Friday, October 31, 2014 4:52:06 PM

This is great but same question as Micheal, how to add anti-malware protection to cloud service?

Arbaz Khan - Tuesday, November 4, 2014 3:36:37 PM

In response to questions from Michael and Arbaz, you can add anti-malware protection to your cloud service by following the documented guidance in the anti-malware whitepaper referenced in the above blog. Currently the support to add anti-malware for cloud services is using API's and PowerShell cmdlets.

Devendra Tiwari - Friday, November 7, 2014 5:19:16 PM

thanks NIC and finally you can only use multiple NICs from Large VM size up.

baomoi - Saturday, November 15, 2014 3:28:33 AM

So, here is an awesome post which can be read with the deal.

Happy New Year 2015 Images Wallpapers Quotes Wishes - Sunday, November 30, 2014 8:38:28 AM

In response to questions from Michael and Arbaz, you can add anti-malware protection to your cloud service by following the documented guidance in the anti-malware whitepaper referenced in the above blog. Currently the support to add anti-malware for cloud services is using API's and PowerShell cmdlets.

7bibty - Monday, December 15, 2014 7:06:55 AM