Chocolatey Professional Edition Review

Software & Operating Systems Windows
, ,
1

Here is my review of Chocolatey licensed, specifically of the Chocolatey Pro edition.

Package vs Software

In this review:

  • A package refers to the Chocolatey managed .nupkg, and/or it’s contents consisting of a .nuspec manifest, installer scripts, and possibly the software binaries.
  • Software refers to the software that a Chocolatey package manages.

Business users notice

This review is of the Professional edition only. I have never used the Business edition. The Business edition does have a superset of Professional features, so it might be expected that this review applies equally to the business edition. However, the Business edition has different pricing, different focus on features, different intended customers, and a different license. Some of the features have different capabilities, for example, the package builder UI has more features in Chocolatey for business than it does in Pro.

Even if the feature I am talking about is the same in both editions, my conclusions are written with a home user’s needs in mind, not business needs.

Also, the Pro edition license does not allow a business to purchase it, business should use one of the other editions. It is permissible for an individual to use a personally purchased Pro license at work.

Table of contents

Feature list of Pro:

More details about the features are available on the Chocolatey.org comparison page

Other items:

Package builder and uploader UIs

Both the package builder and the package uploader are well hidden in the Professional edition. Neither of them have shortcuts created on the desktop or start menu, and they do not have context menu options (unlike the Business edition). However, they are both shimmed into the Chocolatey bin directory, so they can be run with packagebuilder.exe and packageuploader.exe respectively.

The package builder GUI in Pro does not have any of the special features it does in Business, such as silent argument detection or package creation from programs and features. So it basically turns into a replacement for choco new, although one that is more discoverable and easier to initially use. However, it still leaves comments in the nuspec, and due to this and how I work, I normally need to go back and make changes manually to the .nuspec. Such as adding in the Icon URL or details about package parameters. I also always need to go and edit the install and uninstall scripts. So the package builder is nice to have, but does not remove the need for manual editing of the nuspec and scripts.

Instead of using the package builder to help speed creation of new packages, I would suggest instead creating template(s). Templates can allow for a more complete and ready to go package, although most non-business users would probably not be creating enough totally new packages for templates or the package builder to become really useful.

image
image916×688 43.8 KB

The package uploader is a basic GUI for choco push. You give it the path to the package to upload, the URL to upload it to, and the API key for that URL. I suppose it is nice to have the option for a GUI, but since the package sill needs to be packed before it can be pushed, and the pack command is only available via the CLI, there is not much point to using the GUI instead of just running choco push.

image

For both the package builder and uploader, there is nothing really wrong with either of them, just that neither is super useful. I find that I still need to go back and edit quite a few things after the package builder, and the package needs to be packed via CLI before the uploader can be used.

Runtime malware protection

I don’t have much to say about the runtime malware protection, the docs explain it fairly well.

Packages in the community repository are scanned by VirusTotal, and either include the software or have checksumed downloads. Because the VirusTotal integration is based on checksums, the VirusTotal check is pretty much redundant for community packages. And I presume that for any packages created internally you already already trust the software. So, therefore, there is nothing wrong with the VirusTotal integration, but it is but not super useful because generally you would either already trust the software or it has already passed a VirusTotal check.

The generic scanner option is nice, if you want such a thing. You have to manually figure out how to set it up for your particular AV, but once you do then it just works and scans your the packages. It needs the full path to the AV scanner executable, the cmd arguments to give the scanner to scan the selected file, and which exit codes mean that the file is virus free.

Example for Windows Defender. [[File]] is used to tell choco where to intert the path to the file to scan.

choco config set virusScannerType Generic
choco config set genericVirusScannerPath "C:\Program Files\Windows Defender\MpCmdRun.exe"
choco config set genericVirusScannerArgs "-Scan -ScanType 3 -File [[File]] -DisableRemediation"
choco config set genericVirusScannerValidExitCodes "0"
choco feature enable -n virusCheck

Software download CDN cache

The CDN cache just works, at least on supported packages. If the licensed extension sees a that a package install script is downloading a file, it will check if the file is cached on the CDN and if it is, then, it replaces the original URL with the URL of the cached file. It does not do anything if the package includes all the files needed and so does not download anything at runtime.

The cache is fast in my experience, it maxed out a 100mbps down connection. So for anything that might normally have slower downloads (cough cough LibreOffice), it will be sped up.

This cache is useful for packages where the software cannot be included in the package and the download link for the software is the same between versions. It is useful for these packages because they will be periodically broken every time the software updates. This because the checksums change with the new version of the software, and the package has not been updated with the new checksums yet, so Chocolatey will fail to install the package because the checksums don’t match. The cache fixes that, at least for packages that have the cache enabled, because it will intercept the URL and replace it with the cached old version of the software that has the correct checksum.

However, the caching is not universal among packages that have runtime downloads. Companies can opt out of having their software cached, and some package installation scripts are too complex and do not work with chocolatey.org’s caching automation. For example firefox pulls URLs from an included CSV file is not cached.

So this feature works well enough for what is possible due to licensing and other legalities, but is somewhat limited as not everything is cached. To really increase reliability, installation speed, and ability to install old versions, setting up a local nuget repository and doing package internalization / recompilation is what is really needed. There are a number of free options for local nuget repositories. However, internalization is either manual (slow and painful to keep up with), or requires Chocolatey for Business, which has license complications. I am working on scripting of internalizing some packages, but it is still under heavy development.

The EULA for the professional edition, unfortunately not available online, has an regretable clause. It is (paraphrased), that any install benefiting from licensed features counts as as a licensed install. So this applies to items downloaded from the CDN cache, if they were not available from the original URL anymore. This means that the CDN cache cannot be as effectively used to download old version of installers to use on all your machines, because each machine the old install is used on counts against your total of eight machines. So you would need to keep a Chocolatey license active as long as you wanted to use that old installer.

Install directory override switch

The install directory override is very much over hyped by the documentation.

I tried it with a list of 141 packages from the community repository, which is more or less the list I have installed on my main Windows machine.

Of that 141:

  • 29 were packages that don’t have an installation directory per se to override. Examples are; Windows updates, Chocolatey extensions, Chocolatey metapackages, etc.
  • 24 were packages that install into the Chocolatey lib directory. Most of these were .portable packages with a binary that got shimmed onto %path%.
  • 8 were packages that install by extracting a zip, using Get-ChocolateyUnzip or Install-ChocolateyZipPackage. These are planned to be supported in the future, but are not overrideable at the moment.
  • 45 worked correctly and installed in the selected directory.
  • 22 has the extension unable to detect a command line argument that overrides the install directory. These threw an error, which is good.
  • 13 had a command line argument detected, but they did not actually install into the selected directory. This is not good that they failed silently.

That is only a 40% success rate if you count zips and portable packages, and 56% rate if you do not. So pretty much you have to test the overrride for each package before you can use it for real, because it might not work correctly, or, even worse, it might fail silently.

The override does not apply to dependencies, so if you install a package that needs Java, it will override the directory for the package you explicitly installed, but will not override the install directory for Java. This is specifically frustrating in the case of a metapackage. For the example of installing notepadplusplus, the override will not work, because the package will actually install in the dependency package of notepadplusplus.install, which will then actually install the softare. So the the override will not be applied to the softare because it is in a dependency. I can understand why it works like this, but it is not documented well, and tripped me up for a bit.

This is not the only problem, because the override is not easy to use if you want to install multiple packages. It installs the software directly into the the selected directory, and not into a sub directory. So if you select C:\install as the override directory, it will install the software into the C:\install directory, and not into the C:\install\softwareName directory. This means that if you try to install multiple packages, they will all get installed directly into c:\packages, which less than ideal. This means that you have to call choco install once for each package you want to override, and also means that packages.config files are useless if you want to use the override.

It is possible to wrap choco in a PowerShell script or other program to make the override more usable. I have a sample script here in a gist that I wrote for testing purposes of this. But wrapping choco in another program is not really a great long term solution. Fortunately, the chocolatey team seems open to improving this behavior, see my proposal in this GitHub issue.

This all comes together to mean that you first have to check which packages support the override, secondly check which packages have dependencies and then install them first, and finally either tediously individually install each package, or use an external script to do it for you. Due to these limitations, if you want to override directories for a large number of you packages, you have to keep external information about what packages support the override, and what ones need to be installed first due to dependencies. Therefore the global override is not much better then researching what install arguments are needed for the package and passing them manually.

Optimize command/reducer

The Optimize feature does cleanup and removes files leftover after installing a package, for the purpose of reducing disk space usage. It can be run as a command to go through everything, or enabled so as to cleanup after each package install/upgrade. The big draw is that it cleans up larger installers, both the copy extracted into the lib directory and the copy left in the nupkg.

It is very effective at its job. I started with a sample Chocolatey install with about 170 packages which had a lib directory of 11.1gb. Running choco optimize reduced it down to 2.9gb. It also deleted 700mb of files from %temp%. That is a 73% reduction in space used!

However, the freely available choco-cleaner did almost as good a job as the Pro optimize command. It left the lib directory at 3.39gb. The difference between the two is mostly installer exes that choco-cleaner does not remove, unlike choco optimize which does check for and remove them.

However, neither of these should be needed in my opinion. A package manager should not keep extra installer files around, much less two copies. It would be different if they where used as a cache, but they are not, because a new copy is always downloaded by choco upon reinstall.

I think the part that cleans up files in nupkg should be in choco open source, especially since choco-cleaner is able to implement that functionality it in one line of code. Then package maintainers should be instructed and have to clean up large unneeded files that are extracted or downloaded.

Download command

The choco download command is pretty much a worse version of nuget install, at least in the Pro edition. The Business edition has the --recompile/--internalize switch for download which is a nice feature, but the Pro edition does not have that switch.

The download command downloads and unpacks the selected package(s), with optional selection of version and/or source from which to download. However, it is incapable of downloading packages from a packages.config file, unlike nuget install. It also cannot download a package without unpacking it as well, which is annoying if you want to download some packages with which to make a local repository. So you would have to delete the unpacked files, along with waiting for it to unpack them, which kind of hurts the efficiency increase of using choco download. To be fair, nuget install does not unpack the .nuspec, unlike choco download, so choco has the edge in terms of efficiently making changes to a package, repacking it, and putting in a local repository.

Package Throttle

The package throttle limits the maximum download bandwidth that chocolatey can use, both for downloads of packages and for items downloaded during runtime.

I do not see much of a point to this for a home user. A business user may want to kick off updates for dozens of machines at a time programmatically, in which case I can see why a bandwidth throttle could be useful. But for home users, really you would probably be better served by buying a better router that does QoS, which will take care of bandwidth limitations not mearly for choco but for other things as well. And a better router would last for as long as the electronics are function, and not on a subscription model. The other thing you can do is to setup a local nuget repository that caches package from chocolatey.org, which also would take care of bandwidth limitations in most cases.

Chocolatey Automatic Synchronize (Autosync)

The automatic synchronize (autosync) is useful for two things. One is for detecting if software is uninstalled outside of Chocolatey, and then uninstalling the associated package. The other is making the automatic uninstaller more reliable.

Autosync is not useful for tracking software updates and matching them with the package updates. The purpose of the software update tracking is to make the automatic uninstaller more reliable, so that it can find the right GUID and whatnot to uninstall if the software was updated outside of a package update.

With the two functions of autosync in mind, it is a nice feature to have, and I have no complaints.

If a software is uninstalled outside of Chocolatey, you can always remove the package with choco uninstall <packageName> -n --skipautouninstaller in the open source version. And if the auto uninstaller does not work, you can always uninstall the software manually, and run the aforementioned command to remove the package. So Autosync is definitely a nice to have, but not essential feature.

Support

The private email support is one of the best benefits, IMO.

Their support for pro has a 2 business day SLA, which on average they met. I have contacted their support twice with a couple of questions, and had a response time of 10 business hours on average(fairly sloppy counting, just an estimate). There was a variation in response times, some responses where withing minutes, but two others took around a week. The technical level of the support was fine, which is probably caused because they are focused on sysadmins as their main customers, and that they are a fairly small company. When I had a misunderstanding due to some documentation that was ambiguously worded, the documentation was updated within a couple of days.

As to prioritization of bugs and features for customers, it is hard to say, as Chocolatey Pro and `choco` itself have not seen very much change over the past year or so. To be fair, they did release a website overhaul, and a bunch of new stuff for the Business edition. So, it is not as though they have not working, but just not on things for the pro edition.

Other items of note

Choco licensed launches slowly

When the licensed extension is installed, choco starts/launches really slowly. Like painfully slowly, like chromium launches as fast as choco licensed does. I presume that this is because of the extra licensing checks that the licensed extension has, although I do not know.

To get to numbers, here is Choco -h time to run

10 runs total time Single run time
Open Source 4.83 seconds 0.48 seconds
Licensed 18.70 seconds 1.87 seconds
Testing using this script 
$numRuns = 10
[decimal]$time = 0
for ($loop=1; $loop -le $numRuns; $loop++) { $time = $time + [decimal](Measure-Command { choco -h }).TotalSeconds }
write-host 'total time for ' $numRuns ' runs ' $time ' seconds'
write-host 'average time per run is ' ($time / $numRuns) ' seconds'

So Chocolatey licensed takes about three times as long to output the help text.

To give these numbers some context, Chromium was able to launch and exit in about 1.5 seconds, and SumatraPDF was about 1.1 seconds. Those are full GUI applications with decently large codebases, and they launched a full new window, and were still quicker then getting the help text from Chocolatey licensed. For a more apples to apples comparison, running grep.exe --help took under 0.1 seconds, which is over an order of magnitude faster then Chocolatey licensed.

In practice, running OSS choco -h is fine, if sluggish as compared to running other command line utilities. In comparison running licensed choco -h is like running Windows 10 on a fragmented 5400 rpm HDD with two antivirus packages installed on 3gb ram. It is possible that part of this slowness is just part of Windows, and specifically from c#, and I am just spoiled with Linux utilities.

This slowness to start is a big downside to Chocolatey Pro.

Manual license updates

Even though the Chocolatey Pro edition is only available on a limited time basis(1 year), it is setup as through it is perpetually licensed.

There is no tracking in Chocolatey or on the ordering system about when your license is going to expire, so you have to keep track of it yourself. There is no way in choco to find out the license information, so you have to check the XML file manually, or your original email to find out the expiry date. There is a proposal to fix this problem here in this issue.

Then when you want to install a new license, it requires manually coping an XML file into a specific directory, that you may have to manually create. It is possible to do this in a script in a Chocolatey package, which requires extra setup.

The licensing technology is better then a subscription model that requires always online connectivity, or a login server, but is still more work then absolutely needed. It should be a good model for a business user, since they would probably want the flexibility and options of having the license install setup like this.

An upside to the licensing type is that it does not require an internet connection to work. It is possible to set it up on a machine that is completely offline, although of course it initially requires downloading the files on a machine that does have an internet connection.

Conclusion

With regards to Chocolatey Pro features, there are some that are fine, just not very useful to most home users, namely the package throttle and package builder/upload UIs. Then there are the download command and the install directory override, both of which have limitations that make them useless in many circumstances. Finally, everything else is useful to a greater or lesser extent depending you your setup and preferences. So in terms of features, there is no real must-have feature, and nothing that I cannot do without.

Then there is the manual license installation procedure that takes a while, and the slowness of the licensed edition, both of which are drawbacks.

Finally we get to whether you should buy Chocolatey Pro. It is $96 USD/year at the moment, or $67/year if you get the student discount. That is a good chunk of money for a single software product. So if you are thinking of buying it for the software features, I would tend to say no, unless in your particular use case one or more of the features is really really good. However, if getting private support is a large rational, along with the software features of course, then yes, I would suggest it.

Also, there is something I have not mentioned so far, which is that at least some the revenue for the Pro edition goes toward helping keep the community repository running. It is also the only way to provide monetary support to Chocolatey, as they do not have a donations platform. So if you are wanting to donate/support Chocolatey, then this is the way to do it.

6 Likes
2

Damn you went in depth

1 Like