Hacker News new | comments | ask | show | jobs | submit login
How to Distribute Binaries for OS X Using Homebrew (octavore.com)
218 points by tpwong on Feb 15, 2016 | hide | past | web | favorite | 82 comments

> At least on OS X, things are a little easier with Homebrew.

Absolutely. I adore Homebrew and it's ecosystem. Cask, in particular, is quite satisfying; gotta love installing a swath of neccessary fonts from the terminal. And, fwiw, the project is about to be on Homebrew, with no tap required: https://github.com/Homebrew/homebrew/pull/49040 Though getting it up was not quite as pain-free than our initial publishes to NPM.

The thing is: Homebrew solves the problem for just a portion of your audience. In diff-so-fancy's case, we've supported Linux and Windows from the get-go, the only hard part is getting it installed. NPM solves that problem really well, and I can't imagine any other installation technique being our recommendation for Windows/Linux going forward.

This only makes sense if you believe all Homebrew packages are written in JavaScript... which they aren't. Also I've been using Homebrew for about 4x as long as NPM and I have about 1/10th of the issues, so I'll just stick to the package manager that works.

You can use npm to distribute binaries.

That doesn't mean it's a good idea to.

NPM does a great job of distribiting development dependencies and Node.js applications.

It's not really intended for non-js binary builds and platform-specific dependencies.

You can always use NPM to install directly from specific repo release but you'll still have to have to include a native compile-to-platform step somewhere.

Linux Homebrew[0] can cover it for linux. Just need Windows Homebrew.

[0]: https://github.com/Linuxbrew/linuxbrew

Wait, what problem is this solving?

  > Can install software to a home directory and so does not require sudo
  > Install software not packaged by the native distribution
  > Install up-to-date versions of software when the native distribution is old
  > Use the same package manager to manage both your Mac and Linux machines
From the "Features" section of the Github.

It could also likely be used so make a "safe" Linux distro, where the base system is mounted read only, with a writable home partition.

> Cask

In my experience, cask is too fragile for its original use of installing binaries. Issues include

* applications expecting to be installed to /Applications

* applications with weird installation requirements/dependencies

* self-updater conflicting with cask's installation settings

* problems with detecting an installation (and uninstalling it) because the updated version's recipe installs differently or something.

For these reasons I now prefer to install applications by their official/expected/supported installation method.

On the other hand, I find brew very useful.

Using npm to distribute a shell script, seriously? That's not such a good idea. I think what you want is something like the Autotools so your software may be installed with the standard './configure && make && make install'. This way, you make it easy for upstream distributions to package your software.

Wouldn't that have issues with Windows? I'm not a huge fan of NPM but it works really well when you want to be cross platform.

npm only works well for Windows if the program you're distributing is itself in JavaScript using Node. If you're using npm to distribute a shell script, then you still need a Unixy environment for Windows like Cygwin or MSYS, and in that case, autotools is probably no worse.

You don't even need to maintain a separate custom tap repo. Keep the formula in the same repo in a Formula directory, and include the repo URL when tapping (gets rid of the homebrew- restriction)

The brew command for users would be:

brew tap octavore/delta https://github.com/octavore/delta.git

brew install delta


Example above of course doesn't currently work because the formula is not there

(I successfully use it in my repo: https://github.com/stepanhruda/ios-simulator-app-installer)

Keep in mind that Homebrew clones tap repositories when you do 'brew tap'-- this has two major implications if you're keeping your formula in your main source repository:

(1) You're pushing down bits onto users' machines that may not be necessary (like developer documentation, or development versions of your source code if you're developing on master), and

(2) 'brew update' will need to download (via 'git pull') all changes to your repository any time you update, even when the Homebrew formula doesn't change.

Keeping formulas in a separate repository allows Homebrew to work like it's supposed to (and like it does for built-in formulas): an update to your tap repository means there's updates for the formulas that it contains, and you're not downloading any source code except what's actually going to be used, at the time it's going to be used (during 'brew install').

I find the method described in the post to be cleaner UX. Sure, it requires a little extra work on your part to setup, but running a single `brew install octavore/tools/delta` is much nicer and easier to remember than

brew tap octavore/delta https://github.com/octavore/delta.git brew install delta

You're definitely right, just wanted to provide an alternative I found useful. 95% of the time I copy & paste brew commands for custom taps from the repo, I wouldn't remember the 3-part path anyway.

This is probably asking for a PR to hombrew, the ideal command would be

brew install delta https://github.com/octavore/delta.git

A downside of using a tap is that your users subsequent `brew update`s will also have to pull from your repo each time. My current `brew update` has to check 12 separate repos, and takes almost a minute to complete. As a heavy brew user, I'd prefer a packaging convention with less performance impact.

We recently shipped a new updater that parallelizes network requests. The speedup is really appreciable; I have 27 taps and `brew update` now runs under 10 seconds.

Awesome! That's great news

That's awesome, I didn't know about that. I'll update the post with your tip. Thank you!

    > when a post on better diffs showed up on Hacker News
    > last week ... people below are asking why a bash script
    > (that depends on a perl script) is being recommended to
    > install via NPM? [cont. about using Homebrew]
This post is useful in general, but just to circle back on its motivation, for anyone wanting a non-npm (brew) install of the aforementioned diff tool (diff-so-fancy, not made by me) - I've a PR pending merge - Homebrew/homebrew#49040 [0].

In the mean time, you can tap it:

    % brew tap OJFord/formulae
    % brew install diff-so-fancy


>Cross-platform app distribution is a hard problem! [...] At least on OS X, things are a little easier with Homebrew

well.. yeah.. Every OS has an easy way to distribute binaries, the hard part is having something that work on each of them.

Every OS ... except Windows?

Well, there's Chocolatey but that comes with its own set of quirks.

Didn't Windows 10 get PackageManagement FKA OneGet? I mean I haven't upgraded because of privacy, but at least it exists.

Interesting, TIL. I wonder how complete it is. E.g. how much work until you have python and pip? gcc? An SMTP Server? A web server?

Windows Store?

From a user's perspective maybe (although I find the usefulness of that thing highly questionable on a current Windows 10 computer, it doesn't even have a touch friendly official youtube app last time I checked). But from a dev's perspective, i.e. getting the tools to develop something? Nah, not even close.

I've been forcefully switched to OS X recently because work, and overall it's been pretty great (I view it as a well-integrated UI over FreeBSD). A couple of things drive me nuts though:

* (the window manager sucks, but that's not the topic. I had to get it out)

* it needs a good package manager

For us ol' bearded folk (at least spiritually), OS X comes with a slew of unix tools we love. It already includes both vim AND emacs! However, these tools are hopelessly outdated: Despite OS X 10.11 "El Capitan" coming out last fall, Vim is still 7.3 (that's from August 2010. 2010!), and Emacs 22 (2007. That's right.)

So some enterprising folk come to the rescue: Fink, Homebrew, Macports...

Homebrew installs over /usr, so come the next OS update, all that gets squashed. Macports installs into /opt/local, but somehow doesn't survive an OS update. Also, when I tried installing SciPy it went and recompiled GCC4 from source. Guys, the 90s called, and it's been shown that recompiling from source for each user isn't worth the effort.

Apple gave MacPorts some support in the form of equipment, so there's some hints of their preferred method. But the core WTF is this: why doesn't OSX integrate a package manager for its unixy side?

Homebrew installs in /usr/local, not in /usr, and OS updates never touch /usr/local. And if you're paranoid, you can instruct Homebrew to install somewhere else instead.

> OS updates never touch /usr/local.

That's not entirely true. A relatively widespread issue during 10.10 (Yosemite) updates was the installer seemingly locking up and actually moving all the files in /usr/local one by one to then from a backup location veeeery slowly, potentially taking >12h to complete the update rather than a pair of.

I heard people mentioning that (for 10.11 btw, not 10.10), but I never saw anyone give any reason for believing that that's what the installer was doing. Furthermore, I had a pretty extensive /usr/local Homebrew installation and my update to 10.11 completed in the expected time.

Besides, even if the installer did do that, the resulting updated OS still had all the same contents in /usr/local.

As far as I'm aware, the only change OS updates have ever made to the /usr/local folder is resetting permissions back to root:wheel.

> I heard people mentioning that (for 10.11 btw, not 10.10), but I never saw anyone give any reason for believing that that's what the installer was doing.

because that's exactly what the logs showed it doing: https://discussions.apple.com/message/26856483#26856483 and because the preemptive fix (if you hadn't started the update yet) was moving /usr/local's content elsewhere: https://jimlindley.com/blog/yosemite-upgrade-homebrew-tips/

And it was for 10.10. I'm sure people who'd been bitten (or had avoided it through the procedure above) repeated it for 10.11 reflexively or just in case, but the issue was the 10.10 upgrade.

Huh, I only ever heard people mentioning this issue for 10.11. Never heard about it at all when 10.10 came out. I bet that's why I never saw any explanation for this reason, because people who had heard about this with 10.10 were just assuming that's what was going on.

Incidentally, I had no idea about ⌘L to show logs during OS update.

In any case, even if the OS update takes a long time, the end result is you still have the same /usr/local you did before.

Also, FWIW, the discussion you linked to references TeXLive, as does the blog post you linked. I wonder if the issue here is not in fact having anything in /usr/local, but rather having used a package installer to install something into /usr/local (which I assume TeXLive does). It doesn't make much sense to me for the OS updater to move everything out of /usr/local and then back in one-by-one under normal conditions. But if you used a package installer to install into /usr/local, then it would make more sense for it to do something like that (which is to say, it may have special behavior around things covered by package receipts).

Edit: Well, under normal conditions, it might still move the folder away and back, but I'd expect it to just move all the top-level items from the folder back, instead of manually copying all of the nested files).

It's a good idea to install elsewhere, because there are other packages that install into /usr/local (TeX, for example) that aren't in homebrew. I like having my homebrew installation totally separate -- I feel significantly more comfortable upgrading my software to bleeding edge versions if there's no chance of some non-homebrew brain-dead configure script grabbing them. I install to ~/.homebrew, and everything works fine for me. Of course, I don't use ruby at all, or install Python or Emacs packages via homebrew, so I haven't had any of the problems that the homebrew folks warn about on their installation page.

The reason that Homebrew defaults to /usr/local is because there are a lot of braindead packages out there that hardcode the possible locations for software (e.g. /usr/lib and /usr/local/lib), and if you install Homebrew somewhere else, you can't use Homebrew to satisfy the dependencies of any such braindead software you come across.

However, if everything you're installing that depends on Homebrew-provided software is also installed through Homebrew, then it should work regardless of location (I say "should" because I suspect that not every single Homebrew formula has actually been tested with a non-/usr/local install path).

Personally, I stick with /usr/local both because I've never had a problem with it, and because that makes it easier when working with software that uses a default PATH (since /usr/local is usually put in these default PATHs). But if you have any concerns whatsoever, then by all means use a different install prefix.

That's a really weak justification. For most of those "braindead" packages, it is actually fairly easy to get them to install into arbitrary locations. After all, both Fink and MacPort have been doing that with great success for over a decade.

Sadly, you are correct about "not every single Homebrew formula has actually been tested with a non-/usr/local install path". In fact, while there are many Homebrew formulas of great quality, my impression is that there are also m many of rather poor quality, with other problems in them. E.g. just yesterday, I discovered that the Homebreak lmdb formula does not build properly versioned shared libraries -- because upstream didn't, and the packager either didn't notice or didn't care to fix it

I rarely (if ever) this kind of issue with MacPort and Fink, which try hard to avoid such problems (at the cost of a higher entry barrier for packagers -- for better or worse). Mind you, I am not everything is golden with them, either.

Disclaimer: I've been involved with the Fink project since OS X 10.1 or so.

TeX, at least from MacTeX, now installs in /Library/TeX.

Given both the outdated vim/emacs/etc. releases and they've handled the Mac App Store, would you want Apple to be running the platform's main package manager?

Do I wish the 3rd party ones could be better integrated with the OS? Sure. But I'm definitely thankful it isn't Apple calling the shots (or more likely, releasing it and then refusing to call any shots at all for 8 years).

> For us ol' bearded folk (at least spiritually), OS X comes with a slew of unix tools we love. It already includes both vim AND emacs! However, these tools are hopelessly outdated: Despite OS X 10.11 "El Capitan" coming out last fall, Vim is still 7.3 (that's from August 2010. 2010!), and Emacs 22 (2007. That's right.)

And that's way more up to date than it used to be.

> Apple gave MacPorts some support in the form of equipment, so there's some hints of their preferred method. But the core WTF is this: why doesn't OSX integrate a package manager for its unixy side?

Yeah who hasn't dreamt of a package manager handled like MAS.

I'd be happy if Apple helped package managers more, contributed (financially) to them and better took them in account or discussed stuff with them wrt major releases (see: Yosemite homebew woes), but I'm really quite glad the community is completely in charge of macports or homebrew or nix.

What'd be really interesting is if MAS was rebuilt on top of a package manager. That'd solve both the lack of package management, and most of the MAS's woes. (In that you could probably just install MAS apps from the command line, and if nothing else, you could just build an alternate GUI around that.)

> What'd be really interesting is if MAS was rebuilt on top of a package manager. That'd solve both the lack of package management, and most of the MAS's woes.

I don't see how it'd solve the primary MAS woes of horrible contributor/developer experience, uncaring maintainership and inflexible sandboxing.

> horrible contributor/developer experience, uncaring maintainership

I was imagining a package manager with the standard kind of "add whatever sources to your sources list you like" model, as most Linux distros have. If the MAS was rebuilt in terms of this, then presumably adding a custom package source would make custom apps (which, after all, are just packages) show up in the MAS app for selection as well, just as they'd show up in e.g. Ubuntu Software Center. Apple would only control their app/package source.

The interesting question is whether someone would be able to list an app in their third-party source as costing money—and, if so, how that would work, and where payment would go. Maybe each "source" could come with a payment processor token for the MAS client to interact with, additional APIs to let the MAS app grab payment receipts from your source server, etc. In other words, you'd be hosting less of a "package management repo" and more of a "store server with built-in package CDN"—but that's not too big of a change.

Now, I wouldn't expect Apple to actually go this far, but it'd be in line with their current philosophy on enterprise app distribution. Apple really really dislike third-party "app store" apps, third-party "updater" daemon apps, etc.; they would much rather build infrastructure to obviate your enterprise's horrible custom solution than try to have their CSRs understand and support it for you. I could see them building this thing if Microsoft, Adobe, etc. all bugged them to for long enough. And then you as an indie developer could take advantage of it by setting up a PPA-alike (and someone could create the equivalent of Launchpad to host yours and everyone else's.)

Maemo / MeeGo had what you talk about: a store built on top of apt. If I remember correctly, they managed it by adding (on package install through the store gui) package-specific repositories that required authentication. So you could keep using apt-get in the normal way from free repos, receive updates etc.

Aren't MAS apps distributed in a self-contained bundles? What package management is required?

Package management, to me, is mostly about the ability to have a list of sources (feeds of versioned signed package manifests); for updating to generate a merged manifest; and to install, uninstall, auto-upgrade-with-dependencies, and pin/unpin versions of the packages from that merged manifest.

OSX actually has softwareupdate(8), which does work like this—and which at least the OSX system updates (managed now through the MAS app) execute in terms of—which could theoretically be fed custom catalogs (sources) in addition to Apple's, to get most of this part of the benefit. Never heard of anyone doing that, though.

    > the core WTF is this: why doesn't OSX integrate a
    > package manager for its unixy side?
They could, but my guess would be that they don't want friendly-looking tutorials (i.e. ones that the average user wouldn't be scared to follow) saying:

    > Ah, you just need to update your packages; install foo:
    >     macman -Syu && macman -S foo
They don't want to 'have to' offer support for average users accidentally doing 'power user' things, because they were built in.

Except there are lots of other "power user" things that are built in. Where do you draw the line?

It's more likely that they just consider package management out of scope for their part of development-- something that benefits too few people to be worth the cost. Seems to have paid off, too-- package management has been well-handled on OSX for a long time, both through semi-sanctioned projects like MacPorts (which is community-developed but is hosted by Apple) and through community projects like Fink and Homebrew.

    > there are lots of other "power user" things that are
    > built in
Can you think of any that were they not there, they would still implement today?

I thought someone might bring up AppleScript - but I think that's a relic of every user being a 'power user', and personally I really doubt it'd be created today if it didn't already exist; it probably doesn't take much to keep it around now that it is.

I used macports through two or three upgrades before switching to home-brew last year. Switching was one of the best decisions I ever made. In terms of MacOS package managers -- in the grand scheme of things it's not really a big deal. First world problems, right?

MacPorts is very, umm, opinionated. And doesn't seem to trust anything outside of itself. And has some weird opinions, too.

Example: For some reason, svn is a dependency for nmap. Forgot those gravity waves, the next nobel prize belongs to whoever can figure out why a network scanner depends on an obsolete version control system. So macports installs a shiny new version of svn. Which doesn't play nice with the svn that Xcode uses. Overtime I run (macport) svn from the command line it tells me to upgrade my repository, which breaks the Xcode svn.

This kind of stuff is why I abandoned MacPorts. In many cases, I can get by with the system software and I don't need MacPorts to install a duplicate of something. I found that it was taking up a lot of space on my drive. Given that my main drives are relatively small SSDs, I appreciate that Homebrew doesn't require that I switch to a different version of a system tool in many cases.

The worst offender for MacPorts was TeX. I had an official install the latest version from TeX Live and MacPorts wanted to install a second version of TeX. I think they fixed it so you can use a TeX Live version, but it was definitely an issue.

For the window manager, Hyperdock supports snap to half (top/bottom or left/right) and quarter screen. Mission control can be used for virtual desktops.

I use ctrl-left and ctrl-right to switch virtual desktops. Can't remember if that's the default or if I used custom keybinding. There are a bunch of other window management keybindings that can be found in the OSX System Preferences.

Looks like Subversion is required by nmap-update, part of nmap. http://man7.org/linux/man-pages/man1/nmap-update.1.html

They've actually separated out usr/local from all the security stuff leaving Homebrew free and clear for the next OS update

> * (the window manager sucks, but that's not the topic. I had to get it out)

Agree. I really like Spectacle (https://www.spectacleapp.com/) for window management on OS X. As a bonus, it's open source. (https://github.com/eczarny/spectacle)

> the window manager sucks

If you like tiling window managers Amethyst is worth checking out. It's a little unpredictable sometimes but otherwise it works well.

Homebrew sucks on multi-user systems (especially if the Mac in question has previously been single-user) or after the Mac changes its owner/becomes a shared one (company laptop).

Macports handles this usecase far, far better. Give your users sudo rights and be done. Only thing it desperately lacks is allowing third-party additional repositories like Debian's apt does.

You can have additional MacPorts repositories, though I'm sure they're not nearly as configurable as in APT. See, for example, https://guide.macports.org/chunked/development.local-reposit.... Instead of local, the repository you add could be remote. That said, I don't believe I've ever seen a third-party MacPorts repository in the wild.

It's not the default thing to do, but I've found that installing Homebrew under your home directory to work well. Everything's owned by that user, and there are no issues with shared machines (since other users don't see it at all). The downside is that casks (i.e. pre-built installers) might still end up installing things outside, but that would have been the same without Homebrew existing in the first place.

It always bugs me when people don't wipe laptops before handing them off.

In a company environment with shared machines?

I still use macports; what's the current state of opinion of homebrew vs. fink vs. macports?

the current trend, based on what I've seen open-source projects using, seems to be strongly towards homebrew. Not sure if I've seen a project recommending install via fink in quite a few years.

Step 1 use pkgsrc . Not joyent's pkgsrc , but real pkgsrc . Step 2 see step 1.

Step 0 don't use homebrew .

You don’t even need to check the SHA256 by hand, just use `brew create <tarball URL>`. It’ll create a formula with a sensible name, fetch its URL, populate the SHA256, and open the file in your editor. However note it’ll create it in the core formula directory instead of your tap; you’ll need to move it yourself.

>Cross-platform app distribution is a hard problem!

indeed, but couple of things

specific to Mac OS X: there are homebrew, macports, fink and probably more

distributing with homebrew to someone who already going with macports is problematic for example as it is advised to not have both installed.

One alternative is to distribute a pkg and so use PackageMaker to build one.

A pkg has the pros to be native to Apple, but the cons to not work for other systems like Windows and Linux.

So here another alternative: Debian packages (deb).

Wether with macports, homebrew or fink you can install the command line utility dpkg, and using dpkg-deb you can build deb supporting the darwin-amd64 arch.

For Windows, you can use the utility wpkg, http://windowspackager.org/ and with it you can build deb packages for win32 and win64 architectures.

Ultimately if you distribute command-line tools (not GUI app), you can use deb packages for Windows, Mac OS X and Linux Debian (Ubuntu etc.)

Here an example (one of my tools that I need to distrib/install on servers): https://github.com/Corsaair/as3shebang/releases/tag/1.0.0

Debian packages for everyone :)

Linux and Mac OS X install: dpkg -i as3shebang_1.0.0_amd64.deb

Windows install: wpkg -i as3shebang_1.0.0_win64.deb

You can see the build scripts at the root of the repo, they are pretty similar.

Also, once you have deb packages you can also convert them to rpm and other nix-like packages usinga tool like alien, see https://en.wikipedia.org/wiki/Alien_(software).

I guess you have to do this to avoid the name clash with Tigris's delta[0], which is the default if you do `brew install delta`? I'm not sure this is the best solution.

[0] http://delta.tigris.org

I stumbled across this idea a few weeks ago to install some build software on an os x travis ci build machine (using OS X is the easiest way for travis ci to use a C++ compiler that isn't obsolete).

Anyhow, it's nice and simple approach that works.

This is awesome. I was _just_ looking a few days ago for a good tutorial on how to distribute some bash scripts via homebrew and could not find a great resource. Thanks for the writeup.

Rate limit exceeded. http://archive.is/Zp8IQ works.

Whoops, thanks for the heads up! Should be fixed now.

This is still quite a bit more work than publishing to npm:

1. Jump to separate repository

2. Create a .tar.gz with sensitive filename

3. "upload somewhere"

4. Manually update formula, sha, version

4. Push


1. npm bump && npm publish

I ended up automating the formula updating after getting tired of doing that every other day, for a private tap.

I feel your pain, part of the motivation for writing this was that yesterday I wanted to set up a new project, and realized I'd forgotten half the steps. Been thinking that something like https://github.com/aktau/github-release could be helpful for automating the release process, or maybe adding it as a step in a CI workflow.

Its a bit off topic but is there a good alternative for windows? chocolatey doesn't seem to be a good solution.

What don't you like about Chocolatey?


I find OS X's package/dependency management system to be as cobbled together as Windows'.

Linux got it right: https://en.wikipedia.org/wiki/Package_manager

It drives me crazy that Apple removed one the most amazing features of *nix systems: a unified software repository. Brew/brew cask doesn't come close to AUR.

> It drives me crazy that Apple removed one the most amazing features of *nix systems: a unified software repository.

It's not a feature of unices, it's a feature of _Linux_ (Ian Murdoch called it "the single biggest advancement Linux has brought to the industry"[0]).

The vast majority of unices never had a "unified software repository", and BSDs (probably the least indirect ancestor of OSX) have port trees, which at the time OSX was born (ignoring the NeXTSTEP ancestry which predates "modern" BSDs and builds directly from 4.3BSD) were a very different beast and not something suitable proprietary unices (because Apple wasn't going to give the OSX commit bit to randos).

And hell, both apt and rpm were busy _being born_, the premier package managers at the time were raw dpkg and pms(1). And pkgtool[1] I guess?

[0] http://ianmurdock.com/solaris/how-package-management-changed...

[1] http://www.slackware.com/config/packages.php

The problem I'm seeing is that what OS X has isn't a unified system, and what Apple promotes isn't want developers want to/can use. So you see time and effort going into tools like this that only last a few years. It results in more fracturing in the end.

In this popular diagram, https://upload.wikimedia.org/wikipedia/commons/7/77/Unix_his..., NeXTSTEP was originally forked in 1988, before Linux really had package management put together. Darwin is from 2001.

Is 14 years not enough time to establish a proprietor supported unified package management system?

My expectations may just be set too high.

I think you are underestimating Apple's apathy when it comes to doing anything of real interest anymore. We don't even get Darwin .iso releases or changelogs anymore. [0] We are left in an arguably worse state of affairs. It's not just sad; it's boring.

[0]: https://opensource.apple.com/static/iso/

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact