
Microsoft’s Azure DevOps: An Unsatisfying Adventure - jedikv
https://toxicbakery.github.io/vsts-devops/microsoft-devops-ci/
======
ethomson
PM for Azure DevOps here. We've been investing heavily in our user experience
and our CI/CD experience, so I'm sad to see that we've disappointed here.

Some of these complaints I would agree with - in particularly, we're not (yet)
caching build resources - though we're working on this now. But most of these
complaints I was quite surprised to hear; not an experience I would want
someone to have or what I see from the majority of our customers. So this is
certainly a place where I'll follow up with the author to learn more.

~~~
elmalto
We use Azure DevOps extensively at my work and, after having used GitHub,
Gitlab, self hosted solutions, Jenkins, TeamCity... DevOps ranks dead last.

It's the entire experience. The UI is terribly clunky everywhere. The worst
for me are pull requests. Incredibly tough to work with people on a pull
request. I can't even point you to "a" particular problem - for us it's broken
everywhere. From the weird file comparison with overlays to the missing
support for (us) important file formats like ipython notebooks. The way
notifications bubble up in a facebook like format on a pull request. The
emails that have 20000 lines and never the exact information I need. The
amount of clicks to go from comparing two files to open the current version.
Other people mentioned the lacking caching support for builds, so I am not
going to go into more detail here. Another missing piece for me: Where are my
statistics. I'd love to see commit, PR, response, comment etc statistics per
team member / teams / groups.

Using Azure DevOps board is a nightmare of its own...

btw: most of these issues are part of the whole "azure" experience. The Azure
portal suffers from the exact same interface issues of horizontally stacking
fields. One wrong click and your last edits have disappeared into the ether

~~~
sandGorgon
Same here. Azure has a killer feature in India that neither AWS or Google have
- multiple data centres. Because of Indian data residency guidelines, that
becomes critical for business continuity planning.

But the Azure UI is so bad that people are willing to forego stuff like that
just to avoid being on Azure.

The Azure product teams have asked for feedback from startup multiple times
with consistent answers on UI - I'm no longer willing to believe that this
isn't the consequence of some global UI mandate to make things look
"windows-y"

~~~
giaour
AWS has a full region based in Mumbai (ap-south-1) with multiple availability
zones, each of which consists of multiple data centers.

~~~
sandGorgon
umm.. yes we are all aware of this.

Its a single geographic region. Its not allowed according to financial or
healthcare data regulations in India.

P.S. i personally am working with the "Big 4" consulting guys to get a white
paper to work with this. But big missed opportunity for Azure. All because of
bad UI.

------
dustinmoris
Makes me really appreciate my setup of:

\- GitHub for source control

\- AppVeyor for CI

\- BlazeMeter for Load Testing

\- Google Cloud for Cloud Hosting

\- Slack for Team Chat

\- MyGet for private NuGet feeds

\- Trello for task management

\- ELK cloud for log management

Yes it is many different services, but as a small team I like that we can pick
the best tool for the job instead of trying to find a hammer for all nails
which doesn't exist.

It also makes us less worried about changes, because we are not invested in
any vendor so much that if they do something which makes our lives too
difficult that we couldn't easily move on elsewhere.

Also it forces us to design our software in such a way that we don't lock
ourselves into a specific vendor. For example we would have all our builds
scripted instead of configured via some GUIs with the configuration stored on
some vendor's cloud. The build script runs from anywhere. We can run our
builds on Azure DevOps, TravisCI, CircleCi or anywhere. We just prefer to use
AppVeyor because it's the smoothest CI server IMHO, but the transition
elsewhere is just a matter of invoking a single build script.

I genuinely also prefer to work with a product which is doing one thing really
good instead of having one product which does everything mediocre. From an
operations POV it is just as simple as having everything in one place. We have
one browser tab open for each service and it makes us actually a lot more
productive. I am always only 1-2 clicks max away from what I need.

Teams/Companies who choose an all-in-one solution (e.g. everything in Azure +
Azure DevOps) don't care about the productivity of their teams IMHO. They care
more about the one person who has to do the accounting for their subscriptions
or something like that :/

~~~
seanjregan
I was just building a presentation on this topic and the power of an open
approach.

------
markcubed
While I certainly have had a few annoying issues myself, full credit to the
team behind DevOps (nee VSTS) for doing an amazing job over the last >12
months to overhaul and improve the product. I have been impressed at the pace
things are improving.

I have not had the same issues with build machines disappearing into the ether
(we use Windows hosts however) but definitely agree that the (lack of) caching
is maddening. Our apps are not huge but the NuGet & npm restore times now
account for 25-30% of our build time.

~~~
avinium
Yeah, this has been my experience.

I haven't had a build agent disappear, but lack of build caching is very
frustrating. The hosted build agents are slow to begin with, let alone needing
5 minutes for every build to restore packages.

AFAIK there's no way to "only run changed tests" too.

EDIT: Forgot to add that, despite the above, I'm pretty happy with it. The
whole VSTS experience (Wiki/code/PRs/etc) is pretty great, it's nice to have
everything in a single place.

~~~
taspeotis
> AFAIK there's no way to "only run changed tests" too.

[https://docs.microsoft.com/en-
us/azure/devops/pipelines/test...](https://docs.microsoft.com/en-
us/azure/devops/pipelines/test/test-impact-analysis?view=vsts)

~~~
avinium
That looks great - must have overlooked that last time I checked.

------
Someone1234
We use paid Azure DevOps for Source Control (although we're watching GitHub
closely).

We tried to also use Azure Boards (Work Items, Boards, Backlogs, etc). Ouch.
It is a complete UI mess of disjointed ideas. Instead of implementing one
thing well, they implemented two dozen things terribly.

We actually migrated from Azure Boards to Office 365's Planner (KanBan) just
because the UI wasn't so horrible. It is simple and gets out of your way. In
fact O365's Planner, Teams, and OneNote Online products might be superior to
anything Azure DevOps has produced.

The only nice thing I have to say about Azure DevOps Boards is that it
integrates into Visual Studio, but frankly even that is barely passable. I'm
predicting they scrap DevOps completely and just use GitHub as a clean slate.

~~~
tracker1
I don't even use VS if I can avoid it... web dev is so much smoother in VS
Code, and the laginess of VS is unbearable to me. I do have a couple .Net Core
projects I mostly use in VS Code and one MS SQL DB project that I have to use
VS for.

May want to checkout GitLab as an option as well.

------
raxxorrax
Interesting post. I feel for him using this tool. Had my own experience with
team foundation server on premise. Although a different tool, many of the
problems seem quite similar.

A little rant:

To the cloud infrastructure of MS: I don't know if anyone uses it for anything
serious and doesn't complain about severe performance issues not only related
to builds, but everything. From administrative sites to hosted environments. I
read a lot of excuses mainly saying I should adjust my expectations regarding
speed for cloud solutions... Problem is that there are other providers that
don't have these issues.

Seriously, you need only to use office 365 if you want to get an impression of
bad performance. Because everything is slow.

This is from a location in Europe, but I really doubt there are significantly
more performant locations.

~~~
stevenacreman
This is true of all of the services I've tried across Azure, AWS and Google
cloud.

Microsoft are always last place in terms of performance and reliability.

For Kubernetes I actually keep track of exactly how bad they are with an
automated testing tool.

Results of the last test are here: [https://kubedex.com/is-azure-kubernetes-
aks-any-less-terribl...](https://kubedex.com/is-azure-kubernetes-aks-any-less-
terrible/)

I've complained about UX before in the context of the Azure portal being, in
my opinion, terrible. But that's subjective and I respect that others may like
it, although not sure how they do.

~~~
dtrailin
You should do an updated test comparing AWS as well. I feel like there should
be more continuous benchmarking efforts of cloud vendors. This is the last one
I remember in this genre ([https://www.azurefromthetrenches.com/azure-
functions-signifi...](https://www.azurefromthetrenches.com/azure-functions-
significant-improvements-in-http-trigger-scaling/)). I wonder if someone could
make a business doing that.

Also, I've found in terms of UX GCP > Azure > AWS personally.

------
stonewhite
I remember high CPU alarms waking me up, just to see that Azure OMS Monitoring
agents were consuming %99 CPU on every node of our client's database cluster.
I'm not sure why that happened, yet the immediate reaction was to uninstall it
and never use it again.

OP's story about build agents turning unresponsive makes me think of this. MS
is the best cloud vendor to deal as a consulting partner, yet the overall
suffering caused by Azure almost makes me feel like it is not worth it.

------
fastbeef
Look, developer tooling is always going to be hugely dividing. Some will love
it, some will hate it, some will hate it because it's not like the Thing They
Loved At Their Previous Job.

Now, can we talk about the real WTF, which is the name. "Azure Devops",
really? Two extremely potent keywords each on their own? You didn't like
people being able to google answers to their questions about your product?

~~~
tracker1
TFS, VSTS, "Azure DevOps" and often the results are outdated... "ADO" won't
work either. They probably should have stuck with VSTS and just kept the
abbreviation or made it "Azure VSTS" where vsts would at least continue to be
searchable.

Another nicety would be to pay StackExchange to add an Azure DevOps subsite,
assuming there isn't already one.

------
mattbillenstein
How do all of you people get on this broken stuff on Azure with GCP and AWS (I
think) being so good? Is it like free credits?

~~~
throw_away2
I can only assume windows people are used to pain

~~~
kkarakk
pretty much it, also azure is slightly better at pricing in asia

~~~
throw_away2
This mimics the mac vs windows argument I had with my brother-in-law where it
boiled down to the fact that you could run windows on something cheaper than a
macbook. Even though in order to remap caps lock to something reasonable you
have to edit the registry. In 2019. The best answer I think involves windows
server 2003 resource kit and its UI, which is so old it could get a driver's
license.

You save money but what kind of life is that?

~~~
skriticos2
It's not even about the price. I have no problem tweaking my Linux desktop to
use an international Dvorak layout (useful for those European languages with
ä's and é's) and switching the caps and ctrl keys. Say what you want about
Gnome 3, but this works just fine.

On Windows.. I get a US only bare bone Dvorak layout and few ways to remap the
caps/ctrl keys. Messing with the registry is at best annoying on my gaming
(only) PC but impossible on a corporate locked down desktop. Then I'm lucky if
the corporate package repo comes with AutoHotkey that can remap keys in a very
hacky way.

A lot of Windows 98 still shines through in the basic usability suite even on
the newest versions of Windows. Even the on-screen keyboard on a (Android)
phone is more fun than the HID mess in Windows.

~~~
throw_away2
I would love to find an .xinitrc in some dark corner of windows 10 where there
are a bunch of options commented out and you have to pick the right ones or
else your monitor explodes.

This would be an upgrade as at least I would kinda trust that a file I found
on my own system from the vendor can probably be trusted. I trust the options
would be at least mostly self documenting (more self documenting than a random
list of hex digits). And, if I'm going to use software written in the 90s, I'm
going to go with good old vim.

~~~
skriticos2
I love vim, but that .xinitrc stuff is really out of date by now. Gnome 3 now
uses Wayland fairly well and the setup is done with the GUI app Tweaks and the
system settings.

------
codereflection
Tangential: Reading this _really_ makes me appreciate the extremely stable
system my team has build internally with GitLab and TeamCity. Azure DevOps
sounds extremely frustrating (besides the point that I cannot stand the name
they gave this product, and GitLab made that same mistake with the Auto DevOps
feature name).

~~~
sytse
Thanks for using GitLab!

We're open to a better name for Auto DevOps
[https://docs.gitlab.com/ee/topics/autodevops/](https://docs.gitlab.com/ee/topics/autodevops/)
but we haven't been able to find one. Suggestions are very welcome.

BTW We're planning to add per per minute for the CI runners and add shared
MacOS and Windows runners so people that don't want to manage their own no
longer need to.

~~~
codereflection
That's awesome. We have so much customization around TeamCity that moving away
from it doesn't make sense, but we're starting to look at the CI runners for
other things, along with the k8s integration, all great stuff in GitLab. Yeah,
the DevOps name though, it's unfortunate. DevOps is a culture, not a thing,
product, tool, or process. Seems more like calling it Auto CI/CD is more
appropriate, maybe? Anyways, thanks for the response!

~~~
sytse
Thanks for the Auto CI/CD suggestion. I struggle with it since it doesn't
cover things like license management and monitoring.

I agree DevOps is a culture foremost. But at the same time it is starting to
turn into a product category (DevOps tools).

Hope you are able to adopt GitLab with ease.

------
dragonshed
I think title should be updated to "Microsoft's Azure DevOps CI: An
Unsatisfying Adventure", as all the complaints here are about builds but this
service supports lots of other features.

My experience with using DevOps nee VSTS was mostly satisfying. For various
reasons unrelated to the post, my team ended up installing the build agent on
our own bare metal hardware, and other than needing to restart the agent, I
didn't have any mission critical issues. Totally agree with all the UI
complaints.

------
BuckRogers
I use Azure DevOps everyday and like it quite a bit. It'll keep improving but
I've had no complaints. Microsoft is in a very enviable position, not just
having one of the best IDEs but having control over potentially an entire
CI/CD pipeline from Github to Azure, while having one of the major programming
platforms in .Net to enable best-in-class integration. No one else can really
say they have all of that, not Atlassian, Oracle or Amazon.

The CI component, Azure Pipelines, is the only hosted CI tool that supports
all major platforms, at a minimum Windows/macOS/Linux.

The value of essentially fully automating out the role of DevOps engineers is
a long time coming, and I used to do that sort of work but saw the writing on
the wall with things like Azure DevOps. I think a lot of existing DevOps roles
will transition over time to 'Cloud Engineer', specializing in setting up and
maintaining hosted CI/CD platforms. There will also be a lot less of them as a
result. It's becoming a commodity, MS is just one player there. I love what
they're doing and eagerly embrace it, Microsoft has been knocking it out of
the park. It works well for us.

------
Rauchg
We moved to Azure DevOps for Windows testing for multiple large projects
(Next.js, Now Desktop, Hyper) and it's been an absolute pleasure. __Much
__better than our previous Windows CI solutions.

------
devandanger
We are using the on-premise solution with our own build agents to build iOS.
Looking to move actually into Azure Devops to reduce the effort of maintenance
on our own build agents. I've been using AppCenter for hobbyist projects and I
feel its been a good replacement for BuddyBuild. Seems to be pretty reliable
additionally we've thrown a lot of builds at Azure Devops to validate how well
those hosted-agents can handle a large project which also worked out well. So
all in all, we are excited about the hosted-Mac agents as well as the
flexibility to BYO-agent (helps in the transition).

I have seen a lot of these same problems noted in this link as well, just
didn't see to prevent from getting our work done as much. So YMMV depending on
your product/devops use case.

Personally have been using their APIs with some custom CLI scripts to get
around some of these issues.

------
alkonaut
There were some enormous UX issues with the old look & feel of TFS/VSTS as
well. Navigation was nearly impossible to understand, for example.

While I appreciate the work they put into DevOps, it still feels like the same
person that designed the Azure Portal UI has had a say in the UI redesign of
TFS into DevOps and that must either mean that a) Microsoft never realized
what a horrible UX the Azure portal is, or b) they do realize but they are too
invested in this new "design language" to do anything, so instead DevOps
adopted the same style as Azure.

I'm in the process of migrating TO DevOps (on prem) so obviously I have a keen
interest in the direction its taking. Build pipelines being declarative,
navigation being clean up etc. are fantastic improvements. But it still feels
clunky to use, especially around the backlogs/boards.

------
manigandham
VM Agent based builds are an obsolete model. A docker/container-based pipeline
is the current state of the art and allows for very fast builds with caching
built in and isolated stages with narrow responsibilities.

~~~
vtbassmatt
We offer container jobs [1] and recently added the ability to use service
containers [2] as well. Disclosure: I'm a PM for Azure Pipelines.

[1]
[https://docs.microsoft.com/azure/devops/pipelines/process/co...](https://docs.microsoft.com/azure/devops/pipelines/process/container-
phases)

[2]
[https://docs.microsoft.com/azure/devops/pipelines/process/se...](https://docs.microsoft.com/azure/devops/pipelines/process/service-
containers)

~~~
toxicbakery
Funny enough I had not had the time to write about this yet. This resolves one
of the other issues on my list. Thank you.

------
markbnj
> For a few hours, builds ran smoothly. As expected, eventually the agent died
> during a build. Builds sat queued, my frustration grew, and AWS notified my
> instance needed a doctor. After an hour I gave up hope for the unresponsive
> server and commanded it to reboot.

Sounds a lot like some process on the build server is leaking memory. We have
a couple of older services that have this problem, and if we don't catch it
early enough the server will become unresponsive and unable to even handle ssh
connections.

------
kapilvt
As a maintainer on an opensource project thats recently switched out to azure
pipelines, I'm actually overall happy with azure pipelines, its considerably
faster (hosted agent) for us in matrix builds then travis both on latency to
start a build and actual compute power, and cheaper then self hosted drone.
Hosted builds have been reliable for us, we notice a hiccup less than once a
month on a single build (avg 20+ builds a day), minus a service incident
yesterday.

That said I agree with a lot of the criticisms noted here (no retrigger on web
hooks to update status, complicated web ui, log ux, lack build cache). It is
possible to bring your own cache implementation, I just stuffed/restored
things into object storage, also filed an issue for build cache
[https://github.com/Microsoft/azure-pipelines-
tasks/issues/91...](https://github.com/Microsoft/azure-pipelines-
tasks/issues/9190) noting several other _closed_ issues for the same.

It does feel like the platform is moving forward. All that said there's a lot
of complexity and feature set in azure DevOps and pipelines that I haven't
explored and try to ignore in the ui. At the moment its the only cloud
provider build solution that actually even attempts to work well for
opensource projects afaics.

~~~
edglas
Thanks @kapilvt, I'm on the Azure DevOps engineering team. Are there specific
improvements you'd like to see in the logs UX?

------
equasar
At my company, we are pretty happy with Boards, Repos and Pipelines.

One thing that needs to be improved so badly is their Test Plans section, not
so fan for their way to setup agents for Load Testing.

------
rblatz
I've been using this day in and day out for several years now on a ton of
projects. We mostly use hosted git, and pipelines. The lack of NPM cache is a
giant pain, and that's why we run our own build agents. Other than that, I
can't say enough about it. Compared to some of the monsters I've seen built
with Jenkins, I love the simplicity of the builds and releases we have created
in Azure DevOps.

------
nbevans
My mind was blown when we invested a week in setting up DevOps, then realised
it has a critical flaw in that the automated build agent will just go into
hibernation if there was no build activity for a few hours. So then you have
to write a script that invokes a manual build via the API just to keep DevOps
"alive". It's like they forgot to properly setup the IIS Worker Pool timeout
values or something.

~~~
edglas
@nbevans I'm an engineer on Azure DevOps. This is the first we've heard of
this issue, can you provide more details? What error do you see? By
hibernation do you mean Windows hibernation? Feel free to reach out to me at
my HN name at microsoft.com.

------
AdeptusAquinas
I don't know... having been on openshift, Jenkins and bitbucket for the last
year I'd give a large sum of money to go back to azure devops.

------
ryanackley
Came here to say that I've been impressed with Azure DevOps so far. We don't
have the same requirements as the Author. For example, we don't deploy
artifacts to a dependency management system and we don't build on Linux.

I have small complaints here and there but I couldn't find another build
service that has Windows and OS X agents for anywhere near what Microsoft
charges.

------
nikolay
We are actually pretty satisfied and everything's been running smoothly!

------
zeeshanejaz
We migrated from TeamCity and OctopusDeploy a couple of months ago and we are
already regretting it. The reliability of this system is just horrendous. The
documentation is pretty much outdated, and quite a few plugins do not work. I
have to write a bunch of powershell script to get it to work the way I want.
And today, all hell broke loose.

Since this morning, we are stuck with an error message "TF400893: Unable to
contact the server. This is most likely caused by a network error. Please
check your connection and try again.". This has cause a couple of release
pipelines to be get stuck, with cancel option not working either. The paid
"parallel jobs" are being consume in this stuck pipeline and no other project
from my organisation can be built or deployed because of it.

Pretty much regretting having spent so much time configuring the pipelines. We
might switch it doesn't get resolved soon or if it happens even one more time.

------
FlorianRappl
We use Azure DevOps for a fairly large project and so far its working good
(i.e., not great, but better than okay-ish). The point with the lost
communication is something we also experienced more often than I would wish
for, however, on a standard day I do not see the issue appearing at all.

Yes, the mobile page could be optimized (but this is not a deal breaker) and
yes some pages do not subscribe for notifications, but usually all these
things are not in the hot path.

TL;DR: Lots of improvements in the recent year and many things to come - it
may not be perfect but it certainly is usable and provides value to our team.

~~~
toxicbakery
The majority of the product indeed does work pretty well. We make heavy use of
their Boards and Test Plan features and outside of specific but minor UI/UX
complaints, it is a competent tool.

------
Annatar
I'm genuinely curious: what exactly is so hard about installing and system
administering one's own servers inhouse that so many people would put up with
these complications?

~~~
patrickg_zill
There isn't any true difficulty; people are running multiple sites that could
all fit into 1 dedicated 64gb RAM server from Hetzner on a different
infrastructure that allows them to develop their resume...

~~~
Annatar
Are you saying that it's simply a case of resume inflation at employer's
expense?

~~~
patrickg_zill
Sometimes yes. It's not just DevOps but CVops :-)

~~~
Annatar
If I catch anyone at my company doing CVops, there will be hell to pay. Not at
the company's expense and certainly not on my watch!

------
jhacker123
I'm using Azure DevOps (Previously VSTS) more than two years, and meanwhile I
create couple of Pipeline to perform CI/CD.

> .NET MVC Web Application Build and Deploy Using FTP > .NET MVC Web
> Application Deploy in Cloud (Both Azure and AWS) > .NET MVC Web Application
> Build on Linux Visual Studio Agent Create Docker Image and Published

I didn't face any errors from devops, my experience overall good with DevOps.
but i agree that it's not user friendly in term of UI and other workflow.

------
achievingApathy
What I never really understood is why does Microsoft Test Manager only come
with Visual Studio Enterprise? Do only enterprises test their software?

~~~
mandeepj
> Do only enterprises test their software?

Microsoft also has operating expenses. They need to bring home some income to
keep the lights on.

~~~
achievingApathy
It just strikes me as an odd thing to put on a luxury tier like that.
Reiterates that quality and testing is often an afterthought rather than
something everyone should include. If MS wanted the revenue they would put
something the most senior higher paid developers wanted rather than the
testing team.

~~~
darkcha0s
You have plenty of alternatives...

~~~
achievingApathy
True, but I thought the goal of Azure DevOps was to be a complete package
solution. And so while all the pipelines and code and work items are stored
there, it would be a pain to keep test cases in something else. And why no
integration with Excel?

------
jopx
I don't have much complains about DevOps UX, the only problems that sometimes
impact me are the caching/random broken builds.

But Azure Portal UX is another story, the Blade UI is one of the worst things
I've ever used, I hate scrolling horizontally, I can't open things in a new
tab, it's really frustrating.

------
dmarlow
I've yet to look, let alone try, Azure DevOps. There would have to be some
compelling reasons to choose that over TeamCity, which I have had nothing but
great and positive experiences with.

Any big reasons DevOps is something to consider, or is it mostly ideal for new
projects?

------
thejosh
Their build pipeline has been down all morning as well, for both hosted and
self-hosted. Just sits pending. No deploys this morning, woot!

------
Havoc
wow that sounds pretty seriously broken. Has anybody else here actually used
it & can comment?

~~~
GordonS
I've been using Azure DevOps for around 2 years, and TFS for several. Most of
what this article says, I simply don't see in (almost) every day usage.

The single issue I agree with is the lack of caching - installing NPM packages
can take up to 15 minutes in the worst case. MS have known about this for
years, so I honestly don't understand why they haven't done something about
it.

I also think the hosted build machines are underpowered, despite their claims
of Xeon CPUs - build on even an Azure B1ms VM (a low-powered, burstable SKU)
are up to 3 times faster. In fact this is how we solve the NPM caching and
performance issues - by running the VSTS Build Agent (it's OSS!) on relatively
cheap Azure VMs.

Aside from these niggles though, I actually think the Azure DevOps product is
fantastic.

~~~
WorldMaker
Regarding caching, with NPM you actually have a couple good options (best
together):

1.) Make sure you are using `npm ci` instead of `npm install`. The `ci`
install command is often much faster and skips a lot of stuff that `install`
does. It's also better at sticking purely to your package-lock.json versions.
I suggested on UserVoice at one point that Azure Pipelines should switch to
`npm ci` as the npm task default. `npm ci` is a relatively recent npm command
and I don't everyone has caught on to it yet.

2.) You can use Azure Artifacts for NPM packages. It acts as an alternate NPM
feed and caches packages to your Azure DevOps account. Artifacts only supports
NPM and NuGet, so it isn't a general caching solution, but in this specific
case it helps. I've noticed the builds I have using Artifacts are a bit faster
than going straight to NPM. (I started using Artifacts for private NPM
packages, so I'm not using it in every build yet.) The current biggest caveat
to using Artifacts as your main NPM feed for a project is that it doesn't
support NPM audits currently. (I also made sure to report that on UserVoice.)

~~~
Yuioup
I find it weird that in order to fix a problem Microsoft has created (slow
builds due to lack of caching) is to give Microsoft more money. Artifacts
isn't free.

~~~
WorldMaker
Sorry, it's easy to forget what the commodity prices of some of the Azure
DevOps tools are given that various combinations of MSDN licenses and Office
365 plans cover them. It's great that you can pick and choose now, though.

Like I said, caching is only a useful side effect of Artifacts for npm and
NuGet users (that have access to Artifacts). I would believe/hope that
Microsoft may still have interest in adding proper caching to Azure Pipelines
directly.

------
vezycash
It seems no one wants to learn the lesson Facebook was forced to learn - No
wholesale UI changes.

------
ianceicys
At my company we use a mixture of Azure DevOps, Jenkins, CircleCI, Github,
Jira, Artifactory, and a host of AWS and Azure, and my view is that Azure
DevOps (formerly VSTS, formerly VSO, formerly TFService, formerly TFS,
formerly VSS---i kid i kid) is a semi-strong product that just can't get out
of its own way.

The New UI is a clear step backward...and the Github integration leaves a lot
to be desired. If you have Github Enterprise...why use Azure Repos...why are
MSFT teams moving from Azure Repos to Github...and others moving from Github
to Azure Repos. The SCM platform and strategy obviously hasn't settled
yet...so Gitlab is just as viable an offering that's growing more compelling
by the day.

The Azure DevOps hosted build system hasn't been viable in my opinion since it
launched, so on-prem agents are a much better option. Take a look at CloudBees
offerings, they set the poll position.

The key thing that continues to be a pain point is the reporting and project
organization data structure limitations with Azure DevOps: is it 1 TPC per
business unit, 1 TPC for the whole company, 1 team project for the whole
company. What happens when you have 2...how do you consolidate...what's the
path forward if you have to move things around. The answer for the last 8
years has been silence.

The request for a clean solution has been asked for since 2011. Yes 2011, look
at the link below. 8 long years...and basically silence...and "it's a hard
problem"...and multiple updates saying "We will provide an update once we
start planning for the second half of 20XX."

[https://visualstudio.uservoice.com/forums/330519-azure-
devop...](https://visualstudio.uservoice.com/forums/330519-azure-devops-
formerly-visual-studio-team-services/suggestions/2037613-make-it-possible-to-
move-a-team-project-between-te)

Eventually, the promises just ring hollow, the proof is in shipped code, and
in the meantime other platforms and competitors stacks keep building out
better enterprise stories for migrating from Azure DevOps (here's to look at
Github, Gitlab, Atlassian's platforms, Jenkins/CircleCI, TeamCity,
Artifactory, Xebia Labs etc).

Honestly, the world is a MUCH more competitive place with lots of best in
class point solutions that aren't that hard to stitch together and get better
performance/features.

Hope springs eternal...that with the new vertical offerings the DevOps team
can deliver rapid value...but 8 years is a long time to only hear crickets.

If you're starting out and you need a one-size fits all solution and you're a
SMB company with two doze devs...go hog wild with Azure DevOps...that's not to
say it doesn't work for large companies like Shell and Microsoft itself...but
talk with teams about the limitations they run into and make your own decision
on your DevOps transformation adventure...oh and never create more than 1 team
project in Azure Devops...seriously...if you do, game over.

------
whatupmd
Sounds mostly like networking problems with the Azure Cloud.

~~~
toxicbakery
I initially thought the same thing but I'm less convinced of that after trying
an aws based build agent. I think the agents themselves may have a bug that
combined with our pipeline needs, triggers some perfect storm issue. At the
same time, we have seen builds fail to start at all and and sometimes fail to
even checkout from GitHub.

------
lqqkout4elfy
The silver lining here is that Azure DevOps's code review tool is lightyears
ahead of all other repos. I would like to see this support for github repos

------
mcrommert
Thats not 1080p

~~~
toxicbakery
I'm not sure what you mean by this but a few other people have asked me about
the described screen size problem so I decided to add some clarification.

[https://toxicbakery.github.io/vsts-
devops/dimensions/](https://toxicbakery.github.io/vsts-devops/dimensions/)

------
timvisee
I've had similar issues with Microsofts CI implementation. I'll describe some
of the problems I've had here, note that the last time I've used it is about
half a year ago, maybe things have changed though I doubt it. This was with
VisualStudio TeamServices on an instance hosted by Microsoft itself. I
deployed self-hosted build agents as they were orders of magnitude faster.

Probably the biggest problem I've had was that build agents are stateful. The
agent checksout a repository when the build starts, and removes the directory
when it's done. It doesn't however clean up other changes made on the system,
such as cache directories for tooling and other data you might move out of
your build directory. I believe that builds must be reproducible, and
therefore must start with a clean and consistent environment. You don't want
builds to fail down the line because a precious job broke the agents
environment, or even worse, having jobs succeed only on the agent because some
dependent files are left on the agent. Other CI solutions use Docker
containers for this to build in, which is great, as they are immutable (or
stateless, rather) by default. This is in my opinion the single worst
'feature', making these CI builds distrustful.

Secondly, automating these agents is hard. If you want to automatically scale
these self-hosted build agents horizontally based on demand, good luck! When
starting an agent for the first time, you've to go through some cumbersome
configuration wizard to set it up. There is no simple static configuration
file available to use instead, or at least, I wasn't able to find it at the
time. To mitigate the previous problem with stateful agents I created a Docker
container having an agent in it, which was in fact quite difficult to properly
achieve. I automatically restarted these agent containers multiple times a day
in order to clean-up any garbage that was unintentionally left by a job. This
is a horrible hack to slightly improve their (in my opinion) broken system,
but it had the bonus of making these agents more easily scalable.

Another thing is the build configuration. I prefer having a configuration in a
simple YAML file in the same repository, to have it as _close_ to the code as
possible. Although Microsoft does provide such functionality it doesn't seem
to be common to do it this way, and if I remember correctly it's much more
difficult to properly set-up and use than for example, a Travis CI
configuration. They do provide a graphical interface to configure these build
steps though, which is useful when getting started with the system so you can
get a feel of what is possible. It provides some graphical blocks/steps to
achieve various thighs, such as doing an FTP upload or running a CLI command,
with all kinds of weird properties. Maybe that's because Microsofts things are
generally less scriptable. But yet again, I prefer a simple script in an in-
repository configuration file, because it feels somewhat constrained and
limited.

Sadly, it doesn't stop here:

\- sometimes jobs just didn't want to stop, and no, bashing the abort button a
million times didn't fix things. In some situations it took an hour before
jobs were properly killed.

\- the interface (VisualStudo TeamServices) is horrible to navigate, have had
to wild-click many times to find what I need. I'm sure this will improve once
you use it for a longer period though.

\- and there are many other inconsistencies and little problems, such as build
logs loading half of the time.

Maybe I'm grumpy. Maybe I'm ranting about nothing. I don't know. Maybe things
have changed, or will change, at least I hope it will. I didn't have a great
time with their CI system, my experience was horrible as you can probably tell
by now. It feels like an unfinished product, that was released to quickly just
to satisfy customers, so they didn't have to switch to some other platform not
owned by Microsoft for their CI needs. What amazes me is that there are quite
a few free alternatives available which are much simpler and more elegant,
such as GitLab, Travis and Circle CI. They are easier to navigate, are usually
quite quick, don't have the issues I've (and the article have) described. And
greatest of all, you can get it up and running in minutes without having to
worry about it afterwards.

------
CyanLite2
TL;DR: The author works for a Microsoft competitor and doesn't like MS
products.

~~~
fourthark
Strong words. Can you back that up?

I could find no evidence, looks like the guy is an android developer of some
sort.

~~~
toxicbakery
You are correct, I'm just an Android dev but the company I work for is a long
time user of Microsoft products like Visual Studio and obviously Windows. Over
time, if anything, we have become more invested in their products and I don't
see that changing in the foreseeable future as they generally work well for
our use cases.

------
smadurange
This article is nonsense. Azure DevOps is a delight to work with. PR system is
probably the best I've worked with so far. Build pipeline can improve but it's
a far cry from a "horrifically broken".

IMHO, Microsoft has been getting a lot of things right lately. Keep up the
good work.

~~~
toxicbakery
Horrifically broken was an exaggeration, you are correct. Much of this was
written after a stressful day compounded by dealing with our CI failing to do
the thing it is supposed to do. I also agree with you that Microsoft has been
making great strides in the direction of "getting a lot of things right
lately" as you said.

~~~
chrisrpatterson
We really hate to hear you are having such a bad experience. The engineering
manager has reached out on Twitter and included me. I would love to see what I
can do to help

~~~
stevenacreman
I personally don't really understand this approach. In fact, this approach
annoys me.

In this situation I feel like the product has already wasted my time and now
people are contacting me to waste more of my time.

The blog is really detailed and includes screenshots. You could fix the
obvious issues and then follow up with an update once it's released.

~~~
toxicbakery
In the opening paragraphs I stated it was my hope that this gives me a gateway
to communicate with someone that isn't support and isn't the void that is
their robot response issue tracker.

That said, they generously reached out to me and appear to genuinely be
concerned by my experience. I plan to keep posting as I run into issues and
also as they correct my mistakes and also solve my problems.

