
Microsoft’s failed attempt on Debian packaging - LopRabbit
https://www.preining.info/blog/2018/06/microsofts-failed-attempt-on-debian-packaging/
======
ingenieroariel
Having written horrible Debian scripts for open source software that involved
setting up postgres, a java package, some python websites with apache and
seeing the mess I made even with the best intentions [0] I think the article
is too harsh.

Whenever anyone contributes to open source our reaction should be based on
love, not hate, even if they mess up. They will eventually fix it up as
opposed to deciding not venturing into unknown territory anymore.

[0]
[https://github.com/GeoNode/geonode/blob/master/package/debia...](https://github.com/GeoNode/geonode/blob/master/package/debian/rules)

(Edited to fix typo and formatting)

~~~
grumpydba
Sorry but one can never be too harsh when reading "rm /bin/sh" in a shell
script.

~~~
rkangel
Was there ever a stage in your development where you might have done something
like that? This is a rhetorical question because we were all there at some
point - even if it was age 8 with a first computer.

If people are displaying good intentions, and not wildly ignoring a load of
clear instructions, then engaging with them will have a much greater chance of
producing an educated, productive member of the community, and the more of
those the better.

~~~
fnordsensei
Agreed in general, but also, your argument rests on likening Microsoft to an
eight year old using their first computer.

Microsoft surely has the means to hire someone who is not an eight year old
using their computer for the first time.

~~~
rkangel
Microsoft is a very big company, with everything from interns and developers
only with Windows experience through to seasoned _nix developers.

This error is actually a good thing from point of view - it's an anecdatum
that the _whole* company is engaging with open source, not just the bits that
know it well. This is something we should encourage and educate. The advantage
of educating someone at Microsoft as that their institutional knowledge will
persist some fraction of it, so that the next person that comes along is less
likely to have issues.

~~~
seba_dos1
Two words: code review.

------
AceJohnny2
> _What came in here was such an exhibition of incompetence that I can only
> assume they are doing it on purpose._

No. This is how you escalate conflict, not resolve it.

See also: Hanlon's Razor:
[https://en.wikipedia.org/wiki/Hanlon%27s_razor](https://en.wikipedia.org/wiki/Hanlon%27s_razor)

~~~
stcredzero
The vast majority of the time, malice is just an elaborated form of stupidity,
anyhow.

~~~
Gravityloss
That's nice and also fun. Are you quoting someone or is it your original idea?

~~~
stcredzero
I just came up with it, though I wouldn't be surprised if someone else said it
before. Really, it's just recounting my life experience.

~~~
AceJohnny2
because that's what "Hanlon's Razor" states ;)

~~~
stcredzero
Somehow, I'm reminded of a South Park song!

------
pksadiq
I was astonished to see that Microsoft provided this software (R) under the
terms of GNU GPLv2 or GNU GPLv3 explicitly (but not any later version)[0]. May
be because this is a derivative work from GNU R.

That's a good news for free software developers. Microsoft GVFS[1] was using
MIT (while git itself is GNU GPLv2) + some EULA[2]. May be they chose MIT for
GVFS because GNU GPL license explicitly disallows additional restrictions (ie,
EULA can't be added, otherwise it will tbe a violation of GNU GPL).

Edit: I see that the package is distributed as a single binary at
[https://mran.microsoft.com/download](https://mran.microsoft.com/download)
Which shall include (I haven't checked the contents) Microsoft R Open under
GNU GPLv2 and Intel MKL with some EULA (ie, additional restrictions). Won't
that be a violation of GNU GPLv2?

[0]
[https://mran.microsoft.com/faq#licensing](https://mran.microsoft.com/faq#licensing)

[1] [https://github.com/Microsoft/GVFS](https://github.com/Microsoft/GVFS)

[2]
[https://github.com/Microsoft/GVFS/blob/master/GvFlt_EULA.md](https://github.com/Microsoft/GVFS/blob/master/GvFlt_EULA.md)

~~~
DannyBee
No, it's an aggregation which is fine. In more detail: packaging gpl and non
gpl software next to each other is fine, even if your package format is a
single binary.

Linking mkl into gpl software and distributing the binary would not be okay
because you don't have source to mkl you can give people (mkl has binary
blobs). When you link software like this, you need to be able to distribute
the combined binary under gpl, and unless mkl has changed, you can't.

It would be fine to link it and use it locally of course, just not distribute
it.

------
titanix88
This might be off-topic, but time to chime-in with my experience of packaging
an open source software written in python.

Once, I spent a good deal of my time making a grammar-based phonetic
transliterator engine for an Indic language. I was mostly a python developer
with some prior experience in C and plain Make files. After a couple of
rewrites, I was mostly happy with the results. Then was the time for packaging
and making it a usable software. The application needed to install some files
under root so that linux's input subsystem (IBus) could recognize them. Also,
it dependent on some system packages(both python based and non-python based)
like dictionaries for generating suggestions.

As I approached the problem, I learned that I had to learn autotools and a
bunch of very debian specific scripts which were pretty much focused on
packaging applications written in C. It was like an arcane magic. I would
probably had to produce & package at least 10 software to make the return of
investment on that knowledge. I also tried to use python's setuptools, which
felt equally dirty without the benefit of generating a proper .deb package.
The impedance mismatch of the whole packaging architecture and dynamic
language ecosystem was soul crushing.

In the end, I wrote a python script which dumps the project in /opt and
installs/uninstalls the system files and called it a day.

The end result was that user adoption of my software was very low. One reason
for that might be the unusual way of installation(download a tar file, unzip
and run a installer from command line).

~~~
dredmorbius
Did you consult the NMG or ask for assistance?

[https://www.debian.org/doc/manuals/maint-
guide/](https://www.debian.org/doc/manuals/maint-guide/)

Otherwise, your comment lays out the cost/benefit: Debian requires software
fit specific standards. The reward is an unparallelled distribution and
maintenance system.

Note too: there's plenty of python-based software in Debian.

[https://www.debian.org/doc/packaging-manuals/python-
policy/p...](https://www.debian.org/doc/packaging-manuals/python-
policy/python.html)

~~~
titanix88
Yes, I did try to comprehend the NMG, but didn't ask for anyones help. But
AFAIR, back then (mid 2015), the python specific parts of the packaging
manuals were scattered between some presentations and loose documentation here
and there. Thanks for the link though. I might get back to it.

Still, I believe that a robust system like debian could be achieved by more
simpler means. It's packaging system is an invention of a different era and
needs more streamlining to accommodate dynamic language developers. Python was
invented in 90's and it's took 20+ years for debian to setup some serious
guidelines for python developers. What if I want to package a software written
in another dynamic language which was invented a month ago? These questions
need to be addressed.

Disclaimer: I am not claiming that I could device a solution. Just that,
despite my best intentions and best of efforts, I couldn't handle the
difficulty of packaging.

~~~
mmt
> Python was invented in 90's and it's took 20+ years for debian to setup some
> serious guidelines for python developers.

I've only ever packaged in-house (within a company I'm working for) Python
apps, very simple opern-source apps and libraries and modified existing
packaging for new versions of such open source code that otherwise already had
available packaging [1], but my experience has been different to the point
that I disagree [2].

Since I'm a sysadmin and not a developer, my strategy was not just to use the
documentation, but also to look at how other packages handled certain
situations, especially if I knew of an app that was relatively complicated in
terms of build process. For Python, that was mostly a challenge of unravelling
each developer's build/install instructions (easy_install/setup.py/pip) into
something that matched an existing debianized Python package.

> Still, I believe that a robust system like debian could be achieved by more
> simpler means. It's packaging system is an invention of a different era and
> needs more streamlining to accommodate dynamic language developers.

As a sysadmin, I object to the characterization of the debian package system
as being "of a different era", mostly because I don't believe there's anything
about _this_ era, nor about dynamic languages (which have been around for a
very long time anyway) that has eliminated the needs that it addresses.

That robustness is achieved _because_ of the requirement of up-front effort to
conform to a standard that has remained stable across eras.

> What if I want to package a software written in another dynamic language
> which was invented a month ago?

Same as for every other language since the beginning of time: put in the
effort (including asking for help from the community) and conform. If you need
a language-specific standard (such as the OS-default for egg-equivalents),
then make one up (consistent with other languages) and get your new language's
community behind it, if that's possible.

Of course, the other option is just to exist exclusively within that
language's ecosystem (e.g. pip, gem, cpan) and wait for other interested
parties to do the OS-level packaging for you. That happens, too, but it can
slow adoption if your "market" is production environments and not just other
devs in the ecosystem.

[1] Although sometimes with inadequate/inaccurate "debianization" (e.g.
missing expicitly listing a depency)

[2] I do agree that the documentation isn't necessarily organized in the best
fashion, nor that the latest info is in one place, but it's out there. There
might also not be One True Way, but it's arguable if that's a bad thing.

------
maw
That obviously isn't how it should have been done, but the real problem here
is that the packaging system makes it too easy to do this in the first place.

If you use pre and post and preun and postun (rpm terminology) it almost
always is a sign of a deficiency elsewhere, especially when run against files
outside of your package's bailiwick.

~~~
xnyanta
There's nothing wrong with using such scriptlets, especially in the case of
dpkg diversions like the author of this article mentions.

Of course, touching files outside of your package is still a no-no.

------
sirn
Looks like the problem existed (on their RPM package) since 2016[1], notably,
the reason given by their contributor was:

> The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that
> reacting badly with both the R shell wrapper and some Microsoft R Server
> features. I believe we fixed the R shell wrapper features last release by
> explicitly requesting bash and we will be fixing the R server features next
> release. 3.3.3 should be updated with these fixes.

[1]: [https://github.com/Microsoft/microsoft-r-
open/issues/20](https://github.com/Microsoft/microsoft-r-open/issues/20)

~~~
paidleaf
> The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that
> reacting badly with both the R shell wrapper and some Microsoft R Server
> features.

Which is understandable considering the switch from bash to dash as the
default shell has tripped up even open source software.

> I believe we fixed the R shell wrapper features last release by explicitly
> requesting bash and we will be fixing the R server features next release.

So #!/bin/bash?

Doesn't seem like this was a big deal. Just an oversight fixed quickly.

~~~
xenophonf
That shebang isn't portable. Bash isn't always installed in the root file
system. Use this instead:

    
    
      #!/usr/bin/env bash

~~~
rlpb
"#!/bin/bash" absolutely is portable when writing a Debian package for
installation on Debian or Ubuntu systems. bash is guaranteed to be installed
in this scenario as it ships in a package marked Essential.

Further, for the purpose of a distribution package, "#!/usr/bin/env bash" is
wrong, as it'll prefer /usr/local/bin/bash which could be anything. Since the
distribution package uses real package dependencies to express what it needs,
it'd be wrong for it to use anything in /usr/local.

~~~
anothergoogler
No it isn't -- if a user has installed a bash version they prefer, #!/bin/bash
will not use it. Better to trust $PATH and use `env`. It's not as if the `env`
version is more difficult to write.

~~~
rlpb
> No it isn't -- if a user has installed a bash version they prefer,
> #!/bin/bash will not use it.

Correct, but irrelevant for an implementation detail for what is used by a
system package for maintenance scripts where the user will never see what was
used.

> Better to trust $PATH and use `env`.

Wrong, because as we've seen, behaviour can change depending on what /bin/sh
points to. Scripts shipped with distribution packages need to deliver
particular behaviour. The user's choice of shell at the UI level isn't
relevant to this.

> It's not as if the `env` version is more difficult to write.

Irrelevant. The reason for not using it in distribution packages is that it
can lead to incorrect behaviour.

You're arguing with pretty much every Debian package maintainer here. Go look
at distribution package shebangs if you don't believe me.

------
cwyers
For context: this product used to be distributed under the name Revo R Open by
a company called Revolution Analytics. They renamed it Microsoft R Open, but
it's basically the same product: the GNU R distribution with Intel MKL in
place of BLAS, and some custom stuff around working with data too large to fit
in memory. But the package has been distributed for Linux since before
Microsoft bough them.

------
peterwwillis
> I dared to download and play around with the files, only to get shocked how
> incompetent Microsoft is in packaging.

It's not "Microsoft" doing the packaging. It could be an unpaid intern that
volunteered to learn how to make Debian packages.

But that's not even the point: they're _Microsoft developers_. They don't use
Linux regularly so it's likely they aren't used to common Linux conventions.
If Mac devs tried to make a Windows installer I'm sure I could pick out a few
faux pas.

~~~
jessaustin
_...they 're Microsoft developers. They don't use Linux regularly..._

Sure that is my first thought, but we're told that is a prejudice from the bad
old M$ days. Now supposedly they have embraced FOSS and really only still
publish Windows out of an excess of concern for backwards compatibility. Other
than that they're UNIXing all day long over there... although what you say is
certainly plausible.

~~~
WalterGR
_Now supposedly they have embraced FOSS and really only still publish Windows
out of an excess of concern for backwards compatibility. Other than that they
're UNIXing all day long over there..._

Literally no one is claiming anything resembling that.

------
tossaway44
There are similar egregious bugs in the ML Server packages (if you apt
autoremove to clean up old stuff, apt removes the ENTIRE stack because they
have no install checks whatsoever and dpendencies are not handled correctly).

Other teams ao Microsoft (like .NET) seem to have their act together, so I
suppose this is just the R/ML team.

~~~
gsnedders
> Other teams ao Microsoft (like .NET) seem to have their act together, so I
> suppose this is just the R/ML team.

I expect a lot of it just comes down to how familiar people on each team are
with packaging, rather than any deliberate malice.

------
hux_
Ya right. I have wasted many years of my life dealing with Debian/Ubuntu audio
packaging.

Here's the 20 step process for the layman if audio doesn't work -
[https://help.ubuntu.com/community/SoundTroubleshootingProced...](https://help.ubuntu.com/community/SoundTroubleshootingProcedure)

God help you if you are building enterprise software that requires that audio
stack across a hundred different sound cards. Never had any such problem on
Windows.

------
jwilk
If dash was your sh (as it is by default), the dash package will switch
/bin/sh back to dash on upgrade.

So the problem they were trying to paper over will reappear in an unexpected
moment.

------
49bc

        rm /bin/sh
        ln -s /bin/bash /bin/sh
    

What the heck? Installing this package wipes your sh and replaces it with
bash?

~~~
wang_li
Unfortunately for them my /bin/bash is a symlink to /bin/sh.

~~~
mmajcher
Unfortunately for you :)

------
me_again
Discussion on reddit:
[https://www.reddit.com/r/programming/comments/8qb660/microso...](https://www.reddit.com/r/programming/comments/8qb660/microsoft_tries_to_make_a_debianlinux_package/)

------
akulbe
Reading this thread... I'm glad I didn't have this noise around when I was
first learning to program.

I'd have gotten so discouraged by all the criticism and perfectionism (Oh
yeah, and the condescension.) that I figure I'd go hide for good.

Were you all perfect with your code from day one?

------
jancsika
Yes. How insane of Microsoft to mutate global mutable state that Debian's
packaging system blithely lets you mutate.

~~~
rlpb
Debian's packaging system wasn't designed for third parties to ship their own
packages. It has become the norm, but it isn't in Debian's philosophy to
obstruct users from doing what they want.

Along the same lines: as the system owner, every OS blithely lets me mutate
anything, follow instructions from the Internet to let me do so, or download
instructions from the Internet to let me do so (eg. "setup.exe"). The
exception is locked down hardware such as iOS devices.

~~~
jancsika
I'm saying that this article would have been a lot more forceful and
compelling if Microsoft wasn't able to completely bork the author's favorite
OS by installing a single third-party package on that OS.

Compare:

"Holy crap, Foo's web IDE pulls in 20 megs of frameworks and is made up almost
exclusively of eval functions! It takes like 10 seconds to switch to another
tab where I have open a probably still secure online banking session. LOL"

"Holy crap, installing Foo's sloppy third-party package on my OS could
_completely_ bork my OS-- like any other third-party package! LOL"

> The exception is locked down hardware such as iOS devices.

Or a Chromebook, which has an escape hatch for installing whatever less secure
system you want on it, including Debian.

Or Qubes.

Or possibly GuixSD which-- while I'm sure it's not resistant to a bad faith
package-- has user-level installs and transactional updates to recover from an
accidentally borked or sloppy package.

I'm not even dogging Debian. I love and use Debian. And no doubt most other
OSes have similarly brittle system-wide package installers. But the members of
those other OS projects didn't write an arrogant and condescending critique of
a third-party package. A Debian member did. So here we are.

------
ksk
Is there an 'installer' API that package creators can use during installation
instead of sh scripts?

------
djbelieny
rm /bin/sh ??? hard rm a shell from a user's system without any checks or
warnings ? SMH, I can't even...

------
paul_milovanov
s/Microsoft/some guy at Microsoft/g

------
pimmen
Ok, that was pretty awful. Do they want to break stuff on purpose or
something?

~~~
Radle
Obviously not, in every software house there are some people who commit
horrible code.

If you are as big as MS some of it will pass through review.

------
mistrial9
What is the advantage of a dot-deb package ? Authoritative source, reliable
install, signed and secure content.. can we say that ?

Can you imagine the detail and "extras" in a Microsoft legal contract? where
there is money on the table? now compare that to the quality, accuracy and
care shown here.

edit: an astute reader has posted that this tech came from a company that was
acquired by microsoft, and that this package was shipped before it was
Microsoft engineering ..

~~~
dredmorbius
Policy and an exceptionally robust package management system producing
reliable and repeatable results.

[https://www.debian.org/doc/debian-policy/](https://www.debian.org/doc/debian-
policy/)

~~~
marcodave
> reliable and repeatable results.

...given that the environment is in a predictable state. if the package
manager allows for such state to change, and depends on it, how can one be
sure that a package installation/disintallation can be reliable and
repeatable?

~~~
dredmorbius
By making that a requirement of the package maintainer (Debian Developer /
DD).

Debian's installer doesn't thwart errors, Debian _Policy_ (DPM) does. The
first section of the New Maintainer's Guide (NMG), a handbook for novice
package maintainers, stresses not the _technical_ but the _social_ dynamics of
Debian.

[https://www.debian.org/intro/about](https://www.debian.org/intro/about)

[https://www.debian.org/social_contract](https://www.debian.org/social_contract)

[https://www.debian.org/devel/constitution](https://www.debian.org/devel/constitution)

[https://www.debian.org/code_of_conduct](https://www.debian.org/code_of_conduct)

[https://www.debian.org/doc/devel-
manuals#policy](https://www.debian.org/doc/devel-manuals#policy)

There are a few control points within the hierarchy, supreme of which is less
the Debian Project Leader (DPL), but ftp-masters, the (somewhat
anachronistically named) group of people who operate the repository download
sites (mirrors), and are liable for various forms of legal sanction, varying
with local jurisdiction.

Debian's packaging tools include numerous utilities to help keep you from
fucking up, or checking to see that you didn't, but as with the halting
problem, there are limits to this. Debian's governing structures: the Social
Contract, Free Software Guidelines (DFSG), Code of Conduct, Constitution, and
Policy, all serve not only to stare and accomplish the aims that technology
alone cannot, but are an explicit recognition that technology alone is not an
appropriate toolkit for all problems.

Some Debian packages or tools are maintained by teams (security, base, among
others), R seems to be largely or wholly under Dirk Eddelbuettel:

[https://packages.debian.org/stretch/r-base](https://packages.debian.org/stretch/r-base)

Microsoft appear to be either providing freestanding packages or a third-party
repo, which falls outside this framework, though it attempts to use the same
package management tools. There are reasons this is dimly viewed by competent
sysadmins.

[https://mran.microsoft.com/download](https://mran.microsoft.com/download)

TL;DR: If a DD maliciously or incompetently violates policy, they've committed
a policy violation bug. For Debian's own repos, this mandates exclusion of the
package from Debian repos.

