Fear and Loathing - Archives - 2014

Automatically Publishing NuGet packages from GitHub

Tuesday, June 24, 2014

4 Comments

About an hour ago I didn’t have a MyGet account (although I knew about the service) but did have a repository on GitHub with a package that I was manually updating and pushing to NuGet. Thanks to a recommendation from Phil Haack and an hour of messing around with some files I now have a push to GitHub updating my NuGet package with a click of a button. Dead simple. Read on to get your own setup working.

I’m currently in the midst of a reboot of Terrarium, a .NET learning tool that lets you build creatures that survive in an online ecosystem. More on that later but right now I’ve setup a NuGet package that you add to your own creation to get all the functionality of a creature in Terrarium. The problem was is that I was manually editing a batch file every time I built a new version and pushing that build up to NuGet.

This is 2014. There must be a better way. We have the technology. We have the capability to make this process easy.

austin

Thanks to a few tweets and one suggestion from Phil Haack I checked out MyGet. At first it didn’t look like what I wanted to do. I didn’t want package hosting, I wanted to build my package from GitHub and publish it to NuGet. After looking at a few pages I realized that MyGet was really a perfect way to setup “test” packages. Packages I could push and push and push and never make public then with the click of a button publish the version from MyGet upstream to NuGet. In addition to a few pages of documentation on MyGet I stumbled over Xavier Decoster’s blog post on how he does it with his ReSharper Razor extension. The post was good and had enough steps to get me going. The MyGet docs are there but the screenshots and some of the steps are a little obscure (and didn’t always match what I was seeing in my browser) so hopefully this post will clear things up and give you a detailed step-by-step on how to do this (with the current version of MyGet, it may change in the future).

What You’ll Need

You’re going to need a few things setup first for this to work:

A GitHub repository. BitBucket is also supported as well as others but I’ll document GitHub here. Check out the MyGet documentation on what they support.
A MyGet account. I hadn’t signed up for one until about an hour ago so this is easy (and free!)
A NuGet account.
A .nuspec file for your project (we’ll create it in this post)
A batch file to build your package (again, we’ll create it later)

MyGet Feed

You’re going to need to create a MyGet feed first before anything. So once you have an account you’ll add a feed. There are various places on MyGet to add one (and I think once you verify your account you’re thrown into the screen to create one). In the top login bar next to your name you’ll see a file icon that says “New Feed”

There’s also a “New Feed” button in the Your Feeds box on your activity stream page.

Once you click it you’ll be taken to a page where you enter the details of your new feed.

Enter a name for the feed. The first part of the URL is already filled in for you but you specify whatever you want. This will be your URL that you’ll enter into the Package Manager to consume your feed. You can also enter a description here.

Next you get to determine how your feed is to be used. By default it’s public but you can also create a community feed (where anyone can push and manage their own packages on the feed).

With a paid account you can also have a private feed where you invite users to access it.

Click on the Create feed button to finish the setup. Once you create the feed you’ll be dumped to the main page of the package where you can select various settings to configure your package feed. By default there’s nothing there so we’ll build up our package.

Getting Your Code

First thing you want to do is add a Build Service. A build service is a mechanism to tell MyGet how to build your project. This will clone your GitHub repository on MyGet and build your package. Detailed information about the Build Services can be found here.

Click on Build Services then click on the Add build source button. This will allow you to select a build source where you’ll pull your code from.

Here you can see you can add a GitHub repository, BitBucket, CodePlex, or Visual Studio Online service (or you can add one manually).

Choose GitHub and you’ll be asked to authorize MyGet to interact with GitHub (along with maybe signing into GitHub). Once you authorize it you’ll be shown a list of repositories on GitHub you can select from. Choose the one you’re going to build the package from by clicking on the Link? checkbox and click Add (some repositories below are obscured but yours won’t be).

Once you add the build source you now have the information to setup GitHub for the webhook. Expand the build source to show the information MyGet created for it and you’ll see something like this:

Initially you won’t have any builds but this is where you’ll see those builds and their status. What’s important here is the Hook url. This is the value we’re going to use on GitHub to trigger MyGet to fetch the source when a push occurs (and build the package).

Copy the Hook url (there’s a small icon you can click on to copy it to your clipboard, typing is for weenies) and head over to your GitHub project you selected for the build source.

Webhooks

On GitHub in your repository go to Settings then click on Webhooks and Services. Click on Add webhook to show this dialog:

In the Payload URL paste the Hook URL that you got from MyGet. Leave the Content type as application/json and the radio button trigger to just work off the push event then click Add webhook.

This will add the webhook to your GitHub repository which, on a push to the repo, will trigger MyGet to do it’s stuff.

Building

Now its time to setup your package creation in GitHub and create a package. Remember, MyGet is going to be doing the fetching of the source code from GitHub and compiling and packaging. The trigger will be when you push a change to GitHub (you can also trigger the build manually from MyGet if you need to).

By default MyGet will search for a variety of things to try to figure out what to do with your codebase once it gets it. While it does look for .sln files and .csproj files (and even .nuspec files) I prefer to be specific and tell it what to do. The build service will initially look for a build.bat (or build.cmd or build.ps1) file and run it so let’s give it one to follow.

Xavier provides one in his RazorExtensions repository so I (mostly) copied that but he’s doing a few things to support his plugin. Here’s a more generic build.bat you can create that should be fine for your project:

Line 2-5: These lines setup the configuration we’re going to use. This would be “Release” or “Debug” or whatever you have defined in your solution. If it’s blank it will use “Release”.
Line 7-10: This sets up the PackageVersion value that will be passed onto NuGet when creating the package. Again, if it’s blank it will use 1.0.0 as the default.
Line 12-15: Here we’re setting the nuget executable. If you have a specific one in your solution you can use it, otherwise it’ll just use the one available on MyGet (which is always the latest version)
Line 17: This is the msbuild command to run your solution file. My solution file is in a subfolder called “src” so I specify that way, yours might be at the root so adjust accordingly. The %config% parameter is used here to select the build configuration from the solution file. The other values are for diagnostic purposes to provide information to MyGet (and you) in case the build fails.
Line 19-21: Here I’m just setting up the output directories for NuGet. All packages will get put into a folder called “Build” and follows the standard naming for NuGet packages. Right now I just have a .NET 4.0 library but other files may come later. Build up your package folder structure accordingly (the docs on NuGet describe it here).
Line 23: Finally we build the NuGet package specifying our .nuspec file (again mine is in a subfolder called “src”) and the version number. The “-o Build” option is use the Build directory to output the package.

Okay now we have a build.bat file that MyGet will run once it grabs the latest code from GitHub. Commit that (but don’t push yet, we’re not ready). Next up is to build the .nuspec file that will tell NuGet what’s in our package. You can create the .nuspec file manually or using NuGet to scaffold it for you (docs here). There are also some GUI tools and whatnot to build it so choose your weapon.

Here’s my .nuspec for the Terrarium SDK:

Pretty typical for a .nuspec file and pretty simple. It contains a single file, the .dll, that’s in the “lib/net40” folder. A few things to note:

My assembly is in the project directory but the .nuspec file is in the “src” folder so the package contents are relative to where the .nuspec file is, not where the system is running it from (which is always the root with MyGet)
We use the $version$ and $configuration$ token to replace paths and names here. These are set in the batch file prior to invoking NuGet.

Okay now we’ve got the .nuspec to define our pacakge, the source code wired up to notify MyGet, MyGet to get it on a push, and a batch file to create our package.

Push the code changes into GitHub and sit back. You can go to your Build Services page on MyGet for your feed, expand the chevron next to your feed name, and watch an automatically updated display pull your code down and create your package.

If things go wrong feel free to ping me and I can try to help, but realize that everyone’s setup is different. Check the build log for immediate errors. When MyGet fetches your repository it’ll clone it into a temporary directory and then start looking for build.bat and other files to invoke. That’s why we want to put our build.bat file in our root (mine isn’t but I’ll probably change that at some point) so we can get his party started.

A Package Is Born

Once you have a successful build you’ll see your new package under the Packages section.

Cool. You can click on the Package Id to see more detailed information about the package and also manage old releases. Unlike NuGet, MyGet let’s you delete packages rather than just unlist them.

One thing to note here is the version number. Where did that come from? In the screenshot above my version number is 1.8.0-CI0007.

MyGet follows Semantic Versioning (which is a good thing). I do recommend you give it a quick read and consider adopting it. It’s easy and lets people know when things are potentially going to break (if you follow the 3 simple rules about major, minor, and patch version). Semantic Versioning regards anything after the patch number to be whatever your want (pre-release, beta, etc.)

So about that build version. Go back to Build Services in MyGet and for your GitHub source you’ll see a button next to Build called Edit. Click it and it’ll bring up the details about the source that lets you configure how the code is fetched from GitHub.

So down a bit you’ll see a section called Version. Here you can set (or reset) a build counter. Each build on MyGet increments the counter so you know how many times the system was built (you can also use this to track what version is available or how far it’s drifted from source control or different builds in different environments, etc.). The version format field lets you enter your Semantic Versioning formatted value and use {0} as a placeholder for the counter. By default it's set to 1.0.0-CI{0} meaning that your first build is going to be labeled 1.0.0-CI000001 (the build number is prefixed with zeros).

That’s where it came from so anytime you need to bump up the major, minor, or patch version just go here to change it. As for the extra bit we’ll get to it shortly but the default version works and won’t make it’s way into our “production” versions of our package.

Consuming the Package

Now that we have a package we can add it to our package sources in Visual Studio and install it. Again this post is written from the perspective of using MyGet as your own personal repository where you can test out packages before releasing them to the public. While public packages are the default (and free) version on MyGet, you can keep pushing pre-release versions up to MyGet until you’re happy with it then “promote” it to NuGet.

In MyGet click on your package and select Feed Details from the menu. You’ll see something like this:

This is your NuGet feed from MyGet for this package. Copy it (again with the handy icon) and head into Visual Studio.

In the package manager settings add this as a package source. I just add it while I’m developing and testing the package and then disable it (so it doesn’t get confused with the “official” NuGet server). You can easily flip back and forth between packages this way.

I just name the MyGet feed after the package because it’s for testing. Activate the MyGet feed (and to avoid confusion disable the NuGet one) and then return to the managing pacakges for your solution. Select your MyGet feed from the Online choices in NuGet to see your package.

Remember that MyGet is (by default) building a pre-release version (as indicated by the –CI{0} in your version number). This means you need to select the “Include Prerelease” dropdown in the Package Manager:

You should see your package with the same version as MyGet is showing.

At this point you can install your package into a test or client app or wherever you’re using the package. Make changes to it, push the changes to GitHub and magically MyGet will fetch the changes and republish the package for you. Just go into the Updates section in NuGet to see them. Here’s a new package that was created after pushing a change up to GitHub as it appears in the Package Manager GUI in Visual Studio:

Here you can see we have 1.8.0 installed (from NuGet.org). I changed my package source to MyGet feed and there’s a new version available. It’s labeled as 1.9.0-CI00009 (Prerelease).

Like I said, you can keep pushing more versions up to GitHub and go nuts until you’re happy with the package and are ready to release.

Upstream to NuGet

The last piece in our puzzle is having MyGet push a package up to NuGet. This is a single click operation and while it can be automatic (for example pushing everything from MyGet to NuGet) you might not want that. Users would get the pitchforks out and start camping on your doorstep if you pushed out a new version of your package 10 times in a day.

Package Sources are the way to get your feed out. By default one for NuGet.org is even created for you, but it’s not configured or enabled. Click on the Package Sources and you’ll see the NuGet.org one. Click on edit (assuming you have a NuGet.org account and API key).

Enter your information to allow MyGet to talk to NuGet and in the upstream section you’ll see this:

Check the first option. This allows you to push a package from MyGet to NuGet.

Finally we’re ready to go but we don’t want version 1.9.0-CI00108 to be going public. So here’s where Semantic Versioning comes into play. Back in MyGet click on Packages on your package and then click the Push Latest button.

Here’s where the magic happen. MyGet recognizes the “–CI00009” is the prerelease tag in our version. If we leave this in, it’ll push the prerelease up to NuGet. Maybe this is okay but for my use I want to only push up tested packages when I’m ready to release them.

Clear the Prerelease tag field out. This means that will ignore that tag that’s in your version number and only push the version up using the major.minor.patch (in this case 1.9.0 will be pushed up to NuGet.org).

Click Push and in a few minutes it’ll be on NuGet with the newest version.

That’s about it. I know this was long but it was step-by-step and hopefully it should clear up some fuzziness with the process of automatically publishing your GitHub repositories as NuGet packages. As typical it actually took me longer to write this up then to actually do it.

I encourage you to check out more documentation on MyGet as this just scratches the surface as to what it can do and even in this project there are additional options you can leverage like automatically patching AssemblyInfo.cs in your build, labeling the source when you push from MyGet to NuGet, etc.

Feel free to leave your comments or questions below and enjoy!