
How generics were added to .NET - matthewwarren
http://www.mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/
======
ChicagoDave
I started looking at the source diffs and it triggers a panic attack. That's a
ton of work.

All the more reason to admire and appreciate MS Research and the development
of C# generics. Really amazing accomplishment.

And Linq is one of the single greatest inventions in modern day programming.

~~~
btschaegg
Regarding LINQ: I really like certain parts of it, dislike others (mainly
inline Select-statements and the EntityFramework built on top of it), but I'd
be hard-pressed to call it the greatest invention of any time. PEP 255 laid
the ground work for similar language features in 2001, and I doubt that the
python guys "invented" all of that out of thin air (cf. Reference #4 in the
PEP).

What you can argue is that LINQ made this stuff available to more developers
(with useful syntax and in a statically typed language, which also must have
taken a lot of work), but filter/map/reduce have been around for quite a
while.

That obviously doesn't make it less useful or the underlying tech less
impressive though.

~~~
tehlike
The magic behind LINQ is that it's a query layer on top of a backing data
store. When it first came out, you'd see data providers written against
twitter, file system and everything like that. Then entity framework &
nhibernate linq provider came out. It was great, hid a lot of details from
user, brought static type checking to otherwise very hard to "verify" sql.
You'd now think in terms of objects, and not tables.

Linq compiler compiled the whole query into a AST, which you could then
convert to a backing query your data store could understand.

To date, I still think Linq was great. I was indifferent to the "sqlized
linq", but i liked the lambda quite a lot. It was a nice abstraction and a
pretty good language to write queries on.

Disclaimer: I worked on NHibernate Linq integration ~10 years ago.

~~~
brusch64
What is your opinion on the state of NHibernate ?

I've started using it in 2009. I am still using it in my projects, but I am
still not sure if NHibernate is functional complete or if the project is
nearly dead.

It works fine for me (if it is not worth the effort I am using Dapper.net),
but it doesn't seem that there is much active development going on.

~~~
tehlike
To my surprise, it seems fairly active:
[https://github.com/nhibernate/nhibernate-
core](https://github.com/nhibernate/nhibernate-core)

I stopped working on nhibernate around 2010-2011, when i started masters. I
did only keep in touch with one friend - i am not very sociable person. So I
guess short answer is, i am not sure to be honest.

Back in 2008ish, I was thinking it was going to die because Entity framework
provided a great support for linq, and it had a great hype. A bunch of
enterprises moved away from NHibernate (we also had a lot of legacy from XML
configuration - a bunch of internal data structures were referencing xml), so
it was clunky. Fabio, Oren (Ayende) did a bunch of improvements, I worked a
bunch of shortening initialization times and so on. I think Ayende did a very
good job with Linq initially, but there were plenty of edge cases that i
remember having to fix :) It was my first opensource project as a core member
(second and last was castle project) - so it has a very special place for me
:)

~~~
brusch64
Okay - it really seems to be quite active right now. Thank you for your
insight.

I've used the Linq queries in one of my projects, but I like the QueryOver
statements more (a little bit too much black magic in the Linq queries).

But I believe that it was a great effort - so let me thank you for that !

~~~
mattmanser
In my expereince you shouldn't even touch the black magic stuff unless you're
a small org or it's a throwaway app.

Linq with the EF is great for simple queries and updates, anything else and
you're in for serious performance problems as soon as the app scales. Better
to drop to normal SQL for complicated data loading.

Even MS can't write decent LINQ queries, their ASP.Net identity provider is
now the most 'expensive' bit of our app now because they used expensive LINQ
queries instead of using raw SQL queries. Admittedly our use-case is abnormal
as the specific problem we have is that because of the way users are added
their password gets reset almost immediately, as they're invited by an
organiser. It also means new users are constantly being added. When the
password reset is saved it completely unnecessarily "verifies" the update by
making sure the username and email are unique, which means two UPPER()s and
CONTAINS()s on string fields. Unlike the old provider there's no
usernamelowered field already in the db to avoid this.

We can fix it by over-riding these queries, but it's annoying that the ASP.Net
team took a core framework piece that was very performant with fast performing
SQL and made it substandard. At least I can look at the code now ;).

~~~
WorldMaker
I've never thought of LINQ as black magic and have scaled some astoundingly
complex LINQ queries in a past life. I've even done some things in LINQ
optimization that you couldn't do in just "normal" SQL (clever joins of
databases on different servers without linked servers; complex client-side
caching and statistics work).

I think LINQ gets a lot of flak it doesn't necessarily deserve in complex
queries due to people stopping at the black box and assuming black magic. It's
a very functional programming paradigm embedded in an otherwise procedural
world, and so the skills to debug complex LINQ should be unsurprisingly just a
bit different than debugging most else in C#. I don't blame people for
stopping at the black box. I just think more people should know that you _can_
do more than stop at the black box.

(Also, it amuses me that your example from ASP.NET identity's changes have
nothing to with that it should be using raw SQL queries versus LINQ, and
everything to do with denormalization versus the query execution engine. It
shouldn't be a huge surprise that Microsoft might trust SQL Server to be able
to UPPER() and CONTAINS() fast enough, and the queries rare enough that it
isn't necessary to denormalize that information.)

------
zastrowm
This surprised me:

> It was only through the total dedication of Microsoft Research, Cambridge
> during 1998-2004, to doing a complete, high quality implementation in both
> the CLR (including NGEN, debugging, JIT, AppDomains, concurrent loading and
> many other aspects), and the C# compiler, that the project proceeded.

I didn't realize that it was less a "must-have" and more of a "research and
if-possible" task. I wonder how the .NET framework/languages would have
changed if they went with type erasure. Would we even still talking about .NET
today?

~~~
nikanj
We're still talking about Java, which does type erasure. Oracle has really
been phoning it in for many years, and despite that the platform is still
going strong. Killing large, entrenched platforms is not easy.

Windows remains 800lb gorilla on the desktop, and C# is Microsoft's
recommended environment on it. It's hardly surprising that .NET remains
relevant.

~~~
cm2187
On large platforms, I am going to piss off people but I'll do it anyway. It's
not because a language is widely used that it is a well designed language.
Javascript, VBA and php immediatly come to my mind. There are many other
reasons for a language to be a heavy weight other than its own merit.

~~~
eldavido
This reminds me of Peter Lynch's comment about the stock market: "a voting
machine in the short term, but a weighing machine in the long term".

In the short-term, languages with good developer outreach and other factors
win. But you don't get things like generics or the .NET TPL without some
serious long-term vision. I really do believe well-designed languages win over
a long enough timescale.

------
enricosada
Is already at the beginning of the article, but should be more visibible how
MS Research and Don Syme seen the opportunity to add generics (neeeded for F#)
and added that. With design already supporting later addition like variance.

Not just the design, the real implementation too in the existing already
complex CLR codebase. And that include lot of areas like NGEN, AppDomains,
etc, so is no small feat at all, for a production ready framework already used
by lot of developers.

I am programming in .NET Framework from v1.0, the v2.0 (with generics) was a
clear cut, so is not something you can add too late because it become
pervasive in the framework who leverage it in the design (generics in v2.0,
LINQ in v3.0, TPL and Async v4.0) As a note, .NET continue to support non
generic collections for backward compatibility, but usage is deprecated.

Really interesting piece of .NET history in
[https://blogs.msdn.microsoft.com/dsyme/2011/03/15/netc-
gener...](https://blogs.msdn.microsoft.com/dsyme/2011/03/15/netc-generics-
history-some-photos-from-feb-1999/)

~~~
matthewwarren
> Is already at the beginning of the article, but should be more visibible how
> MS Research and Don Syme seen the opportunity to add generics (neeeded for
> F#) and added that. With design already supporting later addition like
> variance.

I definitely agree, that's why I put it right at the start of the post :-)

------
krat0sprakhar
> Ultimately, an erasure model of generics would have been adopted, as for
> Java, since the CLR team would never have pursued a in-the-VM generics
> design without external help.

Can someone explain why "in-the-VM" generics design is better than the erasure
model of Java?

~~~
tehlike
2 main things:

\- Boxing/unboxing would not be an issue anymore, so you gain performance.

\- Runtime types are enforced: List myList = new List<String>();
myList.add(new Object());

In JAVA, this is ok. In c#, it throws.

~~~
eterm
Testing this, it doesn't throw in C#, it just doesn't compile because you
can't declare a variable to be a List without declaring the type with it. If
you specify myList as List<String> then it still doesn't compile.

~~~
tehlike
For c#, try compiling with IList instead of list.

IList myList = new List<String>(); myList.add(new Object());

------
candiodari
Does anybody remember the article about their generics implementation in
assembly ? How they basically were able to reduce the cost of generics by
observing they needed only 3 implementations, by writing specialized versions,
one for 8-64 bit values (just one version), and then one for 64 bit pointers,
and one for structs (struct passed in by having length and value).

~~~
matthewwarren
> Does anybody remember the article about their generics implementation in
> assembly ?

Do you mean [http://joeduffyblog.com/2011/10/23/on-generics-and-some-
of-t...](http://joeduffyblog.com/2011/10/23/on-generics-and-some-of-the-
associated-overheads/) or
[https://blogs.msdn.microsoft.com/joelpob/2004/11/17/clr-
gene...](https://blogs.msdn.microsoft.com/joelpob/2004/11/17/clr-generics-and-
code-sharing/)

They're 2 articles I came across that covered the low-level details of the
generic impl.

------
forinti
Did they really have to split the library into generic/non generic? I think
Java handled that much better.

It's been more than a decade since I last used C#, so excuse me if I recall
incorrectly.

~~~
Maultasche
I disagree. I think that C# generics are way better. The C# generics are
supported at the virtual machine level, whereas Java just pretends to do
generics. Java generic types are unknown to the virtual machine since the
compiler just compiles a List<Foo> to a list of objects, leaving the VM
ignorant of what it really is.

I thought the Java implementation was almost a workaround to avoid the hard
work of a true generics implementation.

The fact that C# generics are understood by all parts of the .NET ecosystem
makes it so much better and boosts the language to a much higher level than
Java.

C# and Java always seemed to be roughly the same to me until C# got generics,
and that boosted C# to a much better plane of existence, which was followed by
a great deal of innovation such as LINQ, async-await, lambdas, etc, all of
which benefit from C# generics.

~~~
xenadu02
To answer the parent's question: Yes, it was necessary. Doing it the way they
did allowed source compatibility with pre-generic or non-generic-aware code.
Passing a List<string> to something that expects an IEnumerable works just
fine and effectively gives you the Java solution (at a source level) when
talking to older APIs! This is why interface IEnumerable<T>: IEnumerable.

>I disagree. I think that C# generics are way better. The C# generics are
supported at the virtual machine level, whereas Java just pretends to do
generics. Java generic types are unknown to the virtual machine since the
compiler just compiles a List<Foo> to a list of objects, leaving the VM
ignorant of what it really is.

It is worse than that. Java generics don't work with value types at all thanks
to this. A C# List<int> is much closer to a zero-overhead abstraction. It does
not box the primitive int values. If Java ever adopts primitive generic types
it will break compatibility or introduce massive performance overhead, as
touching any non-generic API forces a boxing conversion. Either it negates the
reason for type-erasing originally or it eliminates the main benefit of non-
boxing value type support (performance).

(I should clarify that Java doesn't have first-class value type support either
so this really only applies to primitives)

>I thought the Java implementation was almost a workaround to avoid the hard
work of a true generics implementation.

It was a deliberate design decision to retain compatibility between pre-
generic and post-generic Java code.

Personally I think that was the wrong tradeoff; there is never a better time
to make a breaking change than _right now_. The cost only ever increases with
time. It was also clear back then that Java would exist for far longer with
generics than without and that far more code would be written in Java post-
generics than pre-generics. The result is everyone who uses Java is stuck with
limitations and negative performance impacts forever, rather than accepting
some short-term pain.

It is also my personal opinion that source compatibility is what developers
actually cared about and Sun should have told the stodgy risk-averse big Java
houses to suck it up and get ready for JVM v2. It would have been a good
opportunity to fix a few other things in the JVM.

C# had the benefit of hindsight in some ways. I'm not sure if Java
demonstrates how open delivers inferior results, how bad leadership can impact
a project, or just what happens if you pay attention to what "enterprise"
customers claim they want.

~~~
sebazzz
> Personally I think that was the wrong tradeoff; there is never a better time
> to make a breaking change than right now. The cost only ever increases with
> time.

Yes, but from a marketing perspective breaking with the past is usually not
desired. Especially if your product is currently being adopted and is already
partially adopted in both hardware and software (which might have been the
case when they implemented generics?).

------
msie
If it took 6 years to get generics into .NET, we can't fault the Golang guys
for taking their time too.

~~~
josteink
It took them 6 years to do it properly on top of an existing framework with
the aim to maintain a crazy level of compatibility.

They did so because they knew the advantages to be massive. And so does
everyone else at this point.

That golang decided to forego that wisdom when they started from scratch (and
thus create a complex compatibility-story if they were to add it later) is
entirely their own fault.

~~~
msie
So the Golang guys are in the same situation now, aren't they? Isn't it a bit
unfair to fault them for not including generics from the beginning?

~~~
josteink
> Isn't it a bit unfair to fault them for not including generics from the
> beginning?

No. Because that would be much easier and was the only obvious answer.

They took a shortcut, hoped nobody noticed and later when people started
complaining about this obvious omission in a modern language, tried to weasel
their way out of with some “complexity” bullshit and how “people wouldn’t
understand”.

So I can fault them for both not including it and then later being
disenginious about why.

For this glaring omission the golang designers deserve nothing but ridicule.

