1. How to properly do cross-platform, high-performance two-way synced folders between host and guest. Most providers only support one-way syncing. Virtualbox has shared folders, but their performance is pretty lousy and they have issues with relative symlinks. In fact, I still don't fully know what the correct setup is for a dev environment where the files are edited on the host and the guest immediately picks up on them...
2. This idea that with vagrant you'll never again say "It works on my machine" is a lie. So many inconsistencies between Vagrant on Windows, Linux and macOS. Internet connection sharing, line endings issues, symlinking issues, ...
If anyone wants to see how we're using Vagrant:
I don't want to make it sound like Vagrant isn't solving a real problem though, it is. It's just not the unicorn it claims to be.
1. On macOS, host FS volumes are two magnitudes slower. There are a series of hacks that try to mitigate this problem and progress has been made, but it’s still too slow for a dev environment. Linux doesn’t have this problem. I don’t know about Windows.
2. Also on macOS, the host networking bridge doesn’t work the same way it works on Linux. There’s a different hostname that’s used to access the host from within the container that’s inconsistent with Docker on Linux.
Docker is working through these issues, but they’ve been ongoing for years.
I’ve written about Docker dev macOS environments at bradgessler.com, but have given up mostly because FS performance is so poor. Now I spin up what I call “Dockerish” dev environments where my service dependencies run in Docker and the app runs on my host.
As for Windows, it doesn't have the same fs performance issues on volume mounts, but AFAIK file system events have never worked there .
My current reluctant compromise to make volume mounts work seamlessly across platforms is to use polling instead of relying on file system events. That way I can turn on caching on Docker for Mac, and everything just works. The downside is of course polling is inefficient, but I've found when you take care to exclude all non-source directories from the polling, it's usually not as bad as you'd think.
It's pretty easy these days to use tools like Vagrant to spin up the VM then run a Docker Compose file or whatever to get your development environment ready, and would result in identical cross platform environments.
I also like that my underlying code is also within an encrypted VM disk, not laying around on my host mac os.
When I use Reason I'll sometimes mount over SSHFS to edit using neovim in visual studio, though right now I'm building native iOS apps with it so it's actually not in my VM so that xcode can build faster
There's this Vagrant extension  that works pretty well on mac at least (which I actually forked and pushed to rubygems a few years ago when I discovered it because it had been abandoned and no longer worked)
Unfortunately the Vagrant team seems to have no interest in having all 3 of the properties I mentioned just work out of the box, I'm not sure why. Maybe it's just a lack of resources at Hashicorp, I'm not sure how much dev time they put into Vagrant these days since it's otherwise very mature and they have such a large number of open source projects.
But it seems kind of obvious that the functionality needed to get this stuff to work, is an entire network filesystem stack. It seems the obvious choices would be to build into VirtualBox either nfs, or webdav - or leverage host/external support for nfs, cifs, or something like OpenAFS. With sshfs as another pragmatic option. Maybe in the future ipfs will be a viable option too.
But I think cifs is still the "mostly works, with/without complex auth, cross platform" - with webdav a distant second.
I used your vagrant-unison2 plugin for developer environments at Airbnb for a time, although we've now moved on to a more complex, home-grown Unison wrapper with Watchman for filesystem watches.
I set up a TCP server, one light process watches file changes and the other side reacts to it. (Rails). It was one of my silent victories that the entire company used, but I had to do because I was the guinnea pig on the vagrant migration.
On Windows you can register a handler/callback, is that available in Linux?
e.g. tcp mounts, getting read/write blocks matched up btw/client server and sized to be digestable but big enough to move data, etc.
also, nfs is mainly only suited for 'NAS-like' operations - things like rdbms's do waay better on iscsi or eating the vdisk performance.
last I messed with macos nfsd (which has been a while), it a way happier with smaller blocksizes (e.g. 8-64k range) - modern linuces will attempt 1MB which is too much for the older 4.4BSD based code
another thing to look at is timeouts / backoffs - it's easy to kill performance by setting these things too agressively so that the system double-chokes when it gets bogged down..
I did have to supply an extra "mount_options:['actimeo']=1" option to the synced_folder command to make file watcher-type systems work tolerably. With the defaults I'd often be waiting several seconds before any changes made by the host would be picked up by the guest.
Unlike rsync, ZFS always knows what's changed since the last snapshot, so there's no directory scanning.
Unlike inotify, there is (seemingly) no(?) risk of dropped events, which I've experienced with tools like watchman. My ZFS solution still needs prompting event to start the snapshot, but it no longer needs to be told _what_ changed.
Critically the sync flows in one direction. This is a deal breaker for applications that need to communicate file changes back out to the host, but that hasn't been an issue in my use cases.
Unlike NFS, it also means we must (and can) relax the strong consistency model, and so using the filesystem is not so latency bound. This becomes important if the virtual machine is not running locally..
My advice is to cough up an hour or two salary and pay for parallels or VMware (if on Mac)
Vbox (or the Vagrant use of it) is hard coded to use a NAT as the default interface
The common "Vagrant synced folders are slow" relates exclusively to Virtualbox shared folders.
Then there’s the periodic refusal to talk on the network even if the virtual switches are configured properly.
Then the inexplicable “slow boot from hell” which happens randomly where it’ll just hang starting up the kernel at 0% CPU for up to 8 minutes.
This is with CentOS 7 on a gen2 machine.
Honestly though, VMWare is pretty cheap. VMWare Workstation Player and VMWare Fusion aren't cheap upfront, but the upgrade pricing to stay current is decent - I spend more on my JetBrains licenses every year. It's worth it to have a fast and mostly headache-free virtualization experience.
My experience with Vagrant is that it's amazing, but I've only used it for a web application with Github and Jenkins for CI. Dev team doesn't have to spend a second thinking outside the git repo.
11. Shared folders get setup before provision scripts are run
12. You can detect provision has already run (https://github.com/hashicorp/vagrant/issues/936#issuecomment...) using:
14. Virtualbox needs Hyper-V disabled whereas Docker for Windows requires Hyper-V enabled.
I took advantage of this by storing all my settings in YAML and then using the same YAML for both Vagrant and my Ansible provisioner:
Here's the Ruby Script that loads the YAML:
That I call from my Vagrant file:
That is also loaded in all my Ansible roles:
The DHCP client on Ubuntu 16.04LTS doesn't always play nice with multi NIC vagrant machines. All my vagrant boxes are dual NIC (eth0 is the standard 10.0.2.x NAT interface and eth1 is a private interface in the 192.168.56.x range with a static IP - which makes it easier for various vagrant machines to talk directly to each other).
I had an infuriating issue where my boxes would startup and then randomly (it seemed at the time) stop responding to network requests. Initially I thought they were hanging for some reason and would tear them down and re-init them.
Finally thought to enable GUI mode and I noticed that even though a box had stopped responding, I could login fine via the virtual box GUI.
It turned out that Ubuntu's DHCP client was ignoring /etc/network/interfaces (auto generated by vagrant) and wrongly refreshing IP leases on both interfaces (eth0 and eth1).
The trick is to kill the running dhclient during provisioning and restart it with switches that force it to only maintain leases for eth0:
machine.vm.provision "shell", inline: "kill $(pidof dhclient) && /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0"
vb.customize ["modifyvm", :id, "--macaddress1", "auto"]
vb.customize ["modifyvm", :id, "--nictype1", "Am79C970A"]
Anybody know any best practices what to do with Windows developers? I'm thinking of going to docker... but I'm not sure if this really helps.
I always just run a Linux VM and do most things there, with host directories cross-mounted via Cygwin sshd and sshfs (faster, more reliable, and better permissions mapping in my experience than Virtualbox shared folders) and anything that needs to take advantage of filesystem capabilities not supported in NTFS, such as the symlinks mentioned in a sibling comment, done outside a mounted directory.
This lets me get work done, but it's a suboptimal experience to say the least, and relies heavily on my prior systems administration experience to stay working when it goes weird. Absent such experience among your developers or at least someone you can put in support of them, about the best I can recommend is to struggle through with Vagrant and your local handful of best practices - at least with Vagrant your dev environments are reproducible, so that when things go too cattywampus you can just burn down the VM and pop a fresh one out of the oven and get back to work. (If your Vagrant boxes aren't reproducible, that's the first thing you want to fix.)
Docker for Windows is not going to be an improvement here; on the one hand, it's newer and less battle-hardened, and on the other, it has to run in a VM anyway. (Either Hyper-V or VirtualBox, each of which has its own idiosyncrasies, and only one of which can be used at a time - enable Hyper-V, VirtualBox doesn't work any more.) Docker for Windows, like Docker for Mac, comes with some plumbing that tries to smooth over the impedance mismatch between platforms, but it's lossy and brings its own headaches, so you're just adding more complexity to the dev env for no real gain - you can just run Docker in your Vagrant boxes, if you actually need Docker, and have one fewer headache that way.
I usually have a "management" vagrant box that exposes my gitrepo back to the Windows host with samba. I then have two or three other vagrant machines that can access the same gitrepo via an NFS mount to the "management" machine.
This way I can still code in VSCode on my windows host but I do all my git commits via the CLI on the "management" vagrant box. The advantage with this approach is file permissions don't get messed up and symlinks work (i.e. you can follow and edit them on the windows host).
The only major disadvantage is if your repo has symlinks you can't create new symlinks on the windows host and you can't use Git for Windows (and, by extension, the git addon for VSCode). This is because Samba "fakes" symlinks on windows and doesn't truly support them (although it might be able to do so in the future). More info here: https://github.com/git-for-windows/git/issues/1195
That said, anecdotally: repeatedly for the last 3ish years I've been running into docker-on-windows issues that have developed into complete showstoppers during development. This has repeateded across multiple projects and employers. I have an arms-length list of issues that are purely installation related, not even touching my code. I currently am using two dev machines with that had their Hyper-V installations irreparably broken because they were not US-EN windows, rendering them useless for most virtualization. "Full of surprises" is a nice way to put it :)
In those intances, after burning a ton of time, I've landed on a simple conclusion: it is a _lot_ easier to get a VM on OS X to be windows than it is to get windows to pretend to be a competent POSIX environment.
I am optimistic that windows will come around. I am hopeful their posix layer will work out, and their local bash shell will become nice, and the ecosystem will work. I have some faith that next-gen windows will handle this much better...
From experience, though, and for my own projects and developers in 2017: if you want to do container development on windows, get a mac.
It's what I use as my primary development environment for every day web development.
I have a full write on how to get everything working here:
One thing you might want to add is a bit of a warning about the risks of exposing a Docker daemon to the network with tlsverify=false as that would enable anyone who can reach the port on the network to run docker commands and likely take over the host OS.
HyperV should be alright but I don't think it comes with the desktop license for Windows 7+.
VirtualBox never caught up with VmWare when it comes to supported features and stability.
Vagrant is a wrapper around virtualbox so it suffers from the same issues. They could never use VmWare because the free edition doesn't provide API for integrations.
Docker on Windows is totally experimental. They ported to Windows for the PR and next round of funding. Don't expect anything to work.
Same basic idea but plain ruby constants no need for yaml parsing.
The other side of it is that Vagrant is just easier. Docker requires everything to fit into a "one container per service, one process per container" model, which is a really good idea in production, but makes setting up background services (such as the Flow server) in development way harder than it needs to be. I'm not a devops person, so the extra overhead of trying to figure this all out is significant. Vagrant isn't without its own technical problems, but it mostly Just Works™, and I can do anything to the VM that I'd do on a regular Linux machine. As someone who really doesn't care how these things work under the hood, Vagrant has been a significantly smaller source of friction for me.
No, it does not. I use Docker as vagrant replacement too, and use a self-written bash script as "init script" that:
1) launches all services required (e.g. apache, mysql, sshd, elasticsearch) by running "service XYZ start"
2) records the pidfiles of each service (/var/run/xyz.pid)
3) sleeps 10 seconds, waits for all the pids having exited - if yes, exit the initscripts, if not, go back to #3
4) on SIGTERM/SIGINT, gracefully shut down the services in the correct order (e.g. apache/php first, then elasticsearch, then mysql); the watcher loop of #3 will detect all services having shut down and exit cleanly
This way I have exact reproduction of a real target server and don't have to deal with the myriad of issues that arise with docker-compose and "custom networks". Also, the documentation on how to set up a production server, especially the required OS packages, is embedded right in the Dockerfile (or, as I put the setup script in its own file, in this one). Build time for an image with identical functionality is approximately equal to what it was with Vagrant.
In contrast to vagrant, once you build that image initially it starts up way faster (10s for a LAMP+es stack), and provisioning it is a breeze compared to puphpet or whatever is the trend now. And you don't have to update your setup script unless the base OS changes, as compared to puphpet+vagrant - in fact I can use nearly the same setup and init script across stuff as old as Debian 6 (some ancient proprietary software I had to dockerize) over Ubuntu 16.04 and as brand new as Debian nightly.
Having said that, what you did is definitely non-trivial (at least it would be for me), and you've basically re-implemented all of the stuff Vagrant gives you for free, for a pretty marginal benefit (IMO). Maybe that setup works better for you, but I don't understand why I should go through all that effort when Vagrant Just Works™ most of the time, and Docker for Mac runs everything through a VM anyway.
Problem with Vagrant is you can't take the VM you created and deploy it on any random Linux server (or random Docker-hosting cloud provider) - while a pure Docker solution can be deployed literally anywhere with Docker support, as long as you give it a way to persist the data directories of the services. docker-compose is a hit-and-miss across hosters, and you can't use it on DC/OS or Kubernetes environments.
I use my script collection mainly for dev environments, but it's useful when you want to spin up QA/dev instances without having to provision real servers.
- Test my software on a "fresh" Linux installation, to ensure it doesn't have any hidden dependencies
- Test my software with different filesystem layouts and (soon) different filesystems, without having to pollute my own filesystem with hundreds of test files
- Automatically install a set of global commands in the VM (such as "b" to do a complete build) and display help text as soon as a user runs "vagrant ssh", so a new developer can quickly get up to speed
It's been pretty good!
So yes, the author could have run everything on Docker locally, but this was to setup those Docker clusters (Kubernetes and OpenShift).
I did the same thing when experimenting with DC/OS. Running a small cluster on my developer box with 32GB of ram was cheap. Running the same cluster on DigitalOcean gets very expensive:
I know docker gets a lot of hate here but docker4mac is one one the best pieces of software i've used in a my 10 yr career. It totally changed the way i set my dev environments, think about software development/deployment.
I am so glad they are going to support kubernetes now so I can do local-> staging-> prod seamlessly .
I generate my Vagrant files for all my side projects with it and it's a real time saver, especially if you're not savvy in those fancy provisioners or sysadmin in general.
config.vm.define "ums-01" do |machine|
machine.vm.network "private_network", ip: "192.168.1.10"
machine.vm.network :forwarded_port, guest: 22, host: 2210, id: "ssh"
config.vm.define "ums-02" do |machine|
machine.vm.network "private_network", ip: "192.168.1.20"
machine.vm.network :forwarded_port, guest: 22, host: 2220, id: "ssh"
Eventually we replaced it with Dnsmasq and a static IP setup with each development box getting an immutable static IP. Dnsmasq runs on a guest VM that needs to always be up for other purposes as well.
As always the effort from the landrush developers is much appreciated and it may be suitable for a limited number of boxes but it didn't scale with our usecase.
filename:Vagrantfile thing you're looking for
You'd be suprised how many hidden gems are on GitHub that you can use to figure out how to use vagrant. This tip applies more generally.
<stuff you care about> Vagrantfile site:gist.github.com
- I've found VMware Fusion as a provider worth the extra cost/disk space
Maintaining them is admittedly a bit of a time sink though.
Also here's a template for an initial LEMP WP setup
Also very helpful for those using Vagrant on OSX is Vagrant Manager  which gives you menu-bar integration and a quick interface to turn VMs on and off. It's useful even to remind myself when I've left some VMs on, especially if I'm running on battery.
At least one or two of our new hires started with OSX hosts, but switched to Ubuntu after a while, to avoid virtualbox pain.
[edit: added vagrant-lxc link]
I can understand the need for reproducible environments. But when so much time is lost I doubt the first thing you need is something like Vagrant.
To me Vagrant is a tool that you use when the team starts to struggle with "works on my machine". But not before that. Because most of the time (well for PHP at least) it's very easy to make it work on all machines.
We also work on many different projects, often getting dropped into something new without much of a primer. Being able to "vagrant up" and not having to know all the dependencies to get up and running is very handy.
Do we spend time troubleshooting vagrant weirdness? For sure, but compared to the time saved it's a no-brainer.
If you have one dev running MAMP, another running from the default Apple (or Ubuntu/Linux) environment, another running something else, there is no consistency.
We are starting to build simple docker containers for python via RHEL SCL version and will have other docker containers to "play" with for a while. Maybe someday we will get rid of Vagrant. Not likely anytime soon.
If your defaults are currently on "Legacy", empirically we saw 10-30% speedup by switching to "KVM".
Explanation and workarounds: http://www.mihaimatei.com/virtualbox-performance-issues-mult...
Did they fix this?
I mean with a bit of ruby knowhow and some digging you can figure most stuff out, but Vagrant's documentation still lags behind Hashicorp's other stuff which is usually very well documented.
Which is a shame, as it's an incredibly powerful tool, it just lacks the 'last mile' stuff that Docker did so well.
Also, nice username. You'll need a tray.
vagrant-lxc has none of the above problems, it is just containerization with no virtualization penalties.
I've had lots of people tell me that Vagrant sucks. When digging into their problems it's almost always been VirtualBox causing their problems.
I don't use a Linux host so I haven't played with the lxc provider yet but vbox most assuredly does "suck" compared to even the commercial hyper visor alternatives.
I suppose I could install some more of my tools on the VM, but if you take that to its absurd conclusion, I'm just running a clone of my host on the VM.
It seems more useful, maybe, to just run databases and other services in the VM. Those are the more difficult bits to manage usually. A good programming language already has version and package management facilities.
Or perhaps am I missing something? Anyone have another workflow?
In essence, we do most of our development on Windows, but deploy our solution on Linux. For databases, our former development environment relied on Windows drivers that didn't have the same bugs than the Linux implementations. Hence, we caught these bugs much later in the development phase.
Another advantage is that we can deploy different releases of our solution simultaneously.
Of course, the cost to get there was to define a new pipeline to build artefacts that we ship on the boxes, but that was a little price to pay.
I'll be blogging about some of these soon as well
The are quite a few software that use mmap to access files, especially databases. When that happens, either avoid using the shared folder or switch to using the nfs mount.
Here is the ticket about it that's been opened 10 years ago: https://www.virtualbox.org/ticket/819
How would you provide reproducible dev environments?
It will not always replicate exactly what you need on each hypervisor, and so many times you will end up with something not working (example: synced folders with host).
I have always managed my VMs directly in KVM. Takes more time but I don't rely on a middle-layer//wrapper.
If you have well built boxes it shouldn't be an issue.