Hacker News new | past | comments | ask | show | jobs | submit login

Thanks for the link, o for a bunch of new followers on Quora today and was wondering why :)

There are lots of good answers in this thread. In particular, someone took the time to look at the code, and that person wrote it only took them minutes to find similarities that make my opinion difficult to dismiss.

About the lack of documentation, consider that only a tiny fraction of the kernel’s internals surface in the DDK. All our internal headers have a .w extension and are littered with directives like begin_ntddk, end_ntddk (and many others), so a large number of structures and many fields inside existing structures are hidden or replaced by a generic reserved field.

Macros names, parameters, etc. never appears in the compiled code. It is highly improbable (I studied math, “impossible” is not a word lol), almost surely impossible that a clean-room reimplementation ends up using macros for the same things, let alone macros with the same or similar names.

Also, consider that optimizing compilers emit code that is extremely hard to follow even for us. Heck, the debugger, together with the PRIVATE symbols that has the name of all private symbols, and the actual source code, has a hard time to sync with the disassembly with the code when stepping through. While it is possible to write code that behaves similarly seen from outside, it is again implausible that the expression of this behavior results in code that looks nearly identical to the original. Consider that the name of local variables is never part of the binaries, only public symbols are. Also, consider that the compiler aggressively optimizes out variables to reduce memory accesses and holds values in CPU registers as much as possible, so those variables, while conceptually present, don’t really exist as such in the disassembled code. How to explain that a reimplementation comes up with the same variables, declared in the same order, when those variables are optimized out by the compiler?

What about in-line functions, both explicit and auto-unlined by the compiler? How do you even know there was a function in the first place, and how do you invent a name that is identical to the name of said function in the original code?

Funnily, I had a conversation with a very seasoned kernel engineer (I report directly to him) about ReactOS and my Quora reply. He told me the team looked into ReactOS some time ago and reached the exact same conclusions: impossible.

In particular, this person distinctly remembers a hack he implemented (I’m not going to reveal any details, but suffice to say it was in response to some assertion by some 3rd parties that something Microsoft declared in a court of law as very difficult). He explained the hack to me in full details and, boy, hacky that was, and they found the same hack in ReactOS’s code, except that the présumer authors of that “clean room” implementation probably have no idea regarding why the hack was there.

Finally, “clean room” takes another sense when one knows that Alex (yes) worked for Microsoft until spring 2019 at least, as a contractor for a company called Cloudbase Solutions SRL. His Microsoft email address was v-alione@microsoft.com. I don’t know if he had access to the ntos code, or NTFS or anything else, but very close from home he was, for sure.

So to those who want to take my opinion to court, I say it’s a “careful what you wish for” type of thing, but again I’m not a lawyer, and in other regards ReactOS aligns with a very old version of the NT kernel. It is possible (my opinion only) that Microsoft does does not care?




> Consider that the name of local variables is never part of the binaries, only public symbols are.

"Never" and "only public" are wrong in the statement above, because non public symbols were indeed released by Microsoft.

I guess you are young enough not to know that Microsoft accidentally did release some NT builds with the names of the internal variables, and such builds were intentionally made with less compiler optimizations, allowing for easier reversing. Such events of releasing the internal names resulted in some very interesting stories and statements:

https://en.wikipedia.org/wiki/NSAKEY

"_NSAKEY was a variable name discovered in Windows NT 4 Service Pack 5 (which had been released unstripped of its symbolic debugging data) in August 1999 by Andrew D. Fernandes of Cryptonym Corporation."


Also, the Windows code is under shared source license for nearly 20 years. Not really some sensitive secret thing or crown jewels that many want us to believe.


Sure and the source code has also been leaked before. But a "clean room" implementation is not allowed to look at such things.


Back in NT3.x times, for quad-processors machines, and IIRC even NT 4.x with 8-processors machines (like AXEL crazy SMP monsters) Microsoft shipped windows NT as code, and you had to compile it on the machine it was to run on.


Microsoft does not release the kernel’s private symbols, trust me on that. But yes there was some leaks in the past, small portions of NT4 and W2K were leaked, I think I link to a Google query pointing to articles discussing the leaks in the Quora reply.


Well that's categorically untrue. Sure they don't release private symbols intentionally but they have done in the past accidentally. At that point it becomes a bit of a grey area, undoubtedly leaked/stolen source code is a no-no but reversing from private symbols when they do leak seem harder to quantify as you still need to reverse engineer the code, just structures/names etc are already known.

Private symbols are not the only way of gleaning more information, other examples I can think of are:

* Checked builds (prior to Win10). These builds shipped de-optimized kernels (e.g. no inlines) typically with copious debug strings which gave away important details. For example I gleaned a lot of knowledge of ALPC MSRPC from the checked build of rpcrt4.dll from Windows 8.

* SDK/DDK headers, especially in the brave new world of insider previews with preview SDK/DDKs there is sometimes information present which should not have been released including "private" information. Again bit of a grey area.

* The private symbols MS do ship. For example a significant proportion of the COM runtime has private symbols, intentionally. You can extract from those a surprising amount of system call structure information.

I'd recommend watching Alex Ionescu's talk at OffensiveCon about how he does reverse engineering on Windows to see many of these things in action. https://www.youtube.com/watch?v=2D9ExVc0G10

I'm not saying any of this would make it a clean-room re-implementation but to say ReactOS cannot possibly have been reverse engineered without just up and copying source isn't true.

edit: Formatting.


I’d love that you point to an instance where the private symbols of the kernel actually shipped?

It is very possible that some private symbols were part of some leak, but stolen data does not qualify as “shipping” :)

Again, I stand behind my opinion. I eyeballed some of the code side-by-side and there was portions where I could literally see a line-by-line correlation, which I can hardly explain.

Then if reversing the kernel is so doable using legitimate means, why ReactOS is still largely stuck in the early 2000’s, coincidentally where the major leaks happened?


The private symbols have in the past ended up on the public symbol server (and quickly taken down), they have ending up "shipping" in public symbol packs. I can't point to specific incidents as the links to them no longer exist. This is why I said accidentally as they were not released as a conscious effort on MS's part.

However you seem to want to claim the only place those symbols can come from is being stolen. Of course in this case you use leak as a synonym for stolen, bit leak can just as much mean they were released accidentally by the owner, MS can't steal their own private symbols and release them on the web. I'm sure there's some symbol files traded in private scenarios which are actually taken through non public means but there have been actual incidences of public release of private symbols.

I'm not trying to claim that ReactOS is clean, I have no skin in the game from a project or user perspective. For all I know it might have lifted significant portions of its code from stolen source code or the WRK (which isn't stolen in so much as used without permission, which I'd regard as a totally different thing). I do however take exception to the typical software engineer's view there are somethings which cannot be reverse engineered into a almost similar form.

As to why ReactOS is stuck in the early 2000s, it could be because of all the source code which was stolen and put wholesale into the project. Although if that was the case I'd have expect MS would have sued the living shit of the project by now. It could also be because Windows was and is a very complex OS with many layers which if you're trying to re-implement with a team of 10s to 100s versus 1000s it's going to take a lot of time. It's seems unlikely that the project would spend the millions of man hours to create the abomination that is UWP.

Perhaps the best way to determine if ReactOS is unclean is for MS to open source the Windows Kernel, hell why would you even need ReactOS then :-)

edit: Cleanup.


That sounds reasonable (symbols inadvertently leaked). This would be a plausible explanation for at least some of the naming similarities.



The above link points to: “OffensiveCon19 - Alex Ionescu - Keynote - Reversing Without Reversing” and to 9th minute of the talk where it’s very relevant to the current discussion.


Someone pointed to that earlier. Great talk about how he gathers information to write his books.


See again what I quoted up in the thread related to the NSAKEY:

“Windows NT 4 Service Pack 5 (which had been released unstripped of its symbolic debugging data)”


ReactOS goes greater lentgth to preserve backwards compatibility.


Often hotfixes came with private symbols, Microsoft has traditionally been very slack on this.

I’ve seen private symbols for sql server with the guid to switch editions published on the public symbol server for at least 6 months before they were pulled.

Full releases and service packs typically are stripped very well but if you are saying that no private symbols have been published to the public symbol servers then you are incorrect.

The only product that has been effective at stripping symbols traditionally has been office, they were always stripped if you could even get hold of them which was unlikely.

Don’t forget also you could download the checked windows builds which were very open.


> Then if reversing the kernel is so doable using legitimate means, why ReactOS is still largely stuck in the early 2000’s, coincidentally where the major leaks happened?

Because they don't care or need about the newer MS stuff and also don't have the resources either.

Also they can exploit Microsoft good record of backward compatibility, once you have a good enough lower API compatibility, you can just install a lot of newer MS tech directly on top of it.


> Sure they don't release private symbols intentionally but they have done in the past accidentally.

Even if that is the case, it's an incredibly poor idea to use them, so that the code ends up with spurious similarities in spite of being (otherwise) cleanly developed.


Are you arguing that MS mistakenly releasing a version of Windows with debug symbols didn't happen or that it constitutes a leak and isn't fair game?

Because the former is fairly well documented and the latter doesn't seem right in the context of this discussion. If MS themselves messed up and published the symbols through an official channel that's fair game IMO. Although obviously IANAL etc... I'm talking from an ethical perspective, not a legal one.

I don't know much about ReactOS or the NT kernel but we have this type of controversy regularly in the emulation scene and while sometimes it's true that people reuse docs they shouldn't have, a lot of the time people underestimate the skill and cunning of reverse-engineers to figure out how things work without having access to any restricted information.


Kind of late to the game here, but MS could ship actual source code and copying that code would still be copyright infringement. Reverse engineering for inter-operability is legal in many places, but copying implementations is not. Even having seen the code, you would have to reimplement it in a way that worked equivalently, but was different. The OP is claiming that even things like macros have the same implementation and names (i.e. code that is never exposed publicly in any way). Even if you could deduce this from debug symbols (which would be quite tricky, but probably not impossible), you've got to find another way to do that work.

I don't really subscribe to a belief of absolute morality, but in the context of the discussion, I think that no matter how you got access to that code, if you say that you are reverse engineering it, then copying an implementation is not doing what you are saying you are doing (as well as being copyright infringement).


> Are you arguing that MS mistakenly releasing a version of Windows with debug symbols didn't happen or that it constitutes a leak and isn't fair game?

I think he's saying that even having access to leaked or accidentally released originals is not implicit permission to use it freely. Otherwise any piece of software that was ever legitimately released would be fair game, just throw it at a decompiler and profit.

If you're making a clean room design having so many similarities to the original is unlikely to happen accidentally.

Anybody implementing a clean room design should theoretically have no prior knowledge of the original's inner workings. The specs are written by one person, checked to not include any of the original material by a second one, before being passed to a third to be implemented.

From far enough a piece of wire and an isolation transformer do the same thing. The secret sauce is in that isolation, you can't just shunt it and pretend it's the same.

dingby 3 months ago [flagged]

You chuckleheads “released” pretty much everything in the ASSERT statements of checked builds until Windows Vista.


Personal attacks and name-calling, as here and in your previous two comments, are not ok on HN. We ban accounts that do that. It's great if you're providing correct information, but please edit out the swipes.

https://news.ycombinator.com/newsguidelines.html


One small correction. On quora, you state that there is no such thing as public documentation for the NT kernel internals.

There IS public documentation for the NT kernel internals. It is called Windows Internals Book (https://amzn.to/2xCQla5) and has survived its’ 7th Edition


And one of the authors of said book is also quite a prominent member of the ReactOS team.


Did he work for Microsoft though? If yes, I wonder what kind of interaction between him and Mark Russinovich.


No, he did not. Mark Russinovich also started to work on Microsoft only recently.


Mark Mussinovich started working at Microsoft in 2006 according to Wikipedia https://en.wikipedia.org/wiki/Mark_Russinovich


Yep, you are right, https://www.linkedin.com/in/markrussinovich/

But still in 2006 ReactOS already had its 0.3.0 version and 8 years of existence ^) And book Microsoft Windows Internals, Fourth Edition has already been released in 2004


His company got acquired by Microsoft


I cannot edit my comment for some reason, sorry for the typos. About my colleague’s hack don’t get too excited, this is ancient history but it was at the center of a public debate at the time. Some 3rd parties had an ask, the company said it was difficult, this was very true but someone inside came up with a very clever hack that satisfied all parties, the issue was settled and everybody lived happily ever after. It’s just that the nature of the hack relied on very specific implementation details and it’s highly improbably that another implementation would need a hack to begin with, let alone the same hack. It’s one of the many coincidences that I believe will be hard to explain, shall someone really ask.


Most of the kernel code in Reactos is from far before Alex started working at Microsoft. Also the official Microsoft documentation is of course far from the only ‘Windows Internals’ documentation.


Microsoft has leaked tons of private variable names publicly though. See the whole _NSAKEY thing for one example.

https://en.wikipedia.org/wiki/NSAKEY


This is news?

Of course it’s not a proper clean-room copy. This is immediately clear to anyone who’s worked on NTOS or who has source code license.

It been obvious for years. The only possible conclusion is that MSFT doesn’t care.


Anyone spending time giving detailed explanations regarding manual transmissions and engines in general deserves a follow :)


[flagged]


Thanks for the personal attack lol. I watched and did not learn much, except that he’s a quite decent orator. I think what he describes is the process he uses to write his books.


In that talk he literary says how he got names of files and variables.


Why wouldn't you reply to ReactOS devs here pointing that what you are saying is clearly wrong?


What would he say other than what he has already said? The ReactOS devs are not addressing the substantive points. No point in him repeating himself.


Em, for example how the book linked here several times is not documentation on kernel internals?


It’s not documentation no. Some smart people spent a lot of time observing and disassembling and explained a number of things but major aspects are entirely missing, and not all of what is written is accurate. In any case this definitely is not documentation, and certainly not enough information to write not only a compatible implementation, but a compatible implementation with very similar-looking source code.


I hadn't heard of ReactOS but a quick glance at it's kernel source on GitHub and it's OBVIOUSLY straight from Windows.

Not that I work for MS but I used to work on NT graphics drivers and had a copy of the leaked NT kernel source for a while (strictly for personal interest!). This so very like what I remember.

IMHO such blatant theft shouldn't really go unpunished.


I kind of doubt this is the case, because more than 10 years ago they had accusations of this sort and undertook an extensive audit and cleanroom rewrite: https://www.reactos.org/reset-reboot-restart-legal-issues-an...


Microsoft complaining about ripped off source code?

You are the blackest pot in the history of computer software.

https://www.geek.com/news/microsoft-uses-open-source-code-54...

https://lwn.net/Articles/245805/


It's BSD code in both examples. Given that the BSD license explicitly permits the sort of use in question, ripped off seems inaccurate.

Not that microsoft has never crossed any lines - even the naming of the Windows operating system seems morally dubious - but in this case I don't see any issue with what they've done.


It seems like you're personally offended by the project. If some of the code that got presumably stolen is yours I can definitely understand.

Looking at how slowly that project is moving though I think your conclusion is right: They don't care. In some other comment you mentioned how the project seems to be stuck in the 2000/XP area. How relevant does that make this project today? But whether it's based on stolen code or not, it seems like the right choice to focus on getting that completed before moving on to everything that came after. And after all, Windows isn't just the kernel. They have to implement a lot of user space too, because software out there might assume it's available and behaving in very certain ways. Just an explorer clone sounds like a mammoth task on its own.

I still find the project very interesting and overly ambitious, even if code was stolen. I think at their current pace, reactos won't become usable before operating systems became largely irrelevant and everything runs in the cloud. Everyone still working on it today is probably just doing it for the challenge.


Uh-oh. Someone didn't check with MS LCA before posting this. Bad move.


I expressed my personal opinion, here and on Quora. I think I’ll be fine.


Can you please explain what is MS LCA?


Legal & Corporate Affairs


non existent - they are called by a different name in MS (and also have clear guidance for things like this, which was not broken)


Used to be LCA. Now CELA. I left in 2009.


He needs to defend himself in internet court, employment be damned. /s


I find an opinion hard to credit when exact proof is sitting their on an open source repo everyone has access to and a closed source repo only you have access to.

You make a theoretically compelling argument like the "hack" you describe above and then leave out the only part that really matters where you prove it. It is entirely likely that you are simultaneously entirely correct and entirely unable to prove it due to company policy but this leads us to the next logical question.

If you knew you couldn't prove it why did you open your mouth? It seems the only result will be negative PR.


It appears you did not read what I wrote. The similarities are out there for anyone to see, there is plenty of leaked Windows code floating around including on GitHub, and I did not need anything else to form my opinion.


This was flagged for some reason. Please don't flag comments that don't violate community standards just because you don't like the author.

I have no intention of reading through millions of lines of code from a legally questionable source to make your point for you.

How about you do so and post an in depth analysis?


Do you think senior management at MS believes your comments are a good idea or at best benign? Or do you think all the details of your comments have just not yet fully reached VP level, legal, and PR?

I'm thinking they'd disapprove. That's just a guess, maybe they think it's fine and great. However, it may be worth at least considering another possibility.

Does your company want random technical employees, making non-technical statements, about how people are "ripping off" MS?

It's your job to raise the flag internally and help people understand up and down the chain, and across functional lines the context and perceived severity of the issue.

It's your job to write a blog post about how great your containers are architected. Not be a lone wolf, self appointed spokesperson using inflammatory language on issues with potential legal implications.

Please don't take this comment in the vein of you're just an engineer shut up and code, in fact you have a critical role as someone with domain expertise.

The point is when you stumble onto a high visibility cross-functional issue, I've found on sensitive topics many organizations seem to appreciate when someone reacts by coordinating, discussing, and facilitating a cross functional, unified response with one designated voice, that acts in the best interests the company.


Why are you policing other company's employees and presuming to know what their job is?


First off I'm not policing anything, because, how would I have the authority to do that?

Secondly you object to me presuming I know what his responsibilities are. I don't claim to know what he codes. However I'm able to make a pretty good guess about what the scope of a MSFT senior developers job is and this kind of thing is not even close, unless it's a special case worked out in advance with others. It wasn't worked out in advance in this case because he says that he made the comments before getting feedback from more senior people on his team.

Finally I assume some of the down votes have mistakenly conflated my comments with trying to hide information behind corporate walls.

I advocate hiding nothing - and fully support the new generation of companies who believe in transparency and ethics.

Being transparent and ethical has nothing to do whatsoever with letting developers try and make legal decisions when it's not their area of expertise. It also doesn't mean that a company shouldn't work together across departments to try to decide the right thing to do.

Good management and coordination between roles don't preclude transparency or doing the right thing in anyway shape or form .


This was a fairly pedantic post but I'll reply anyways.

> First off I'm not policing anything, because, how would I have the authority to do that?

Maybe policing isn't the right word, but your earlier post was fairly condescending to axelriet. "I think they'd disapprove... It's your job to do this... It's your job to do that..."


It's subjective, but let's be real, you're probably right.

Moreover, I don't believe in such a philosophy between any two people, so to the extent it is I regret the tone of the comment.

If it makes any difference, the true source of the tone was based on thinking similar to, holy crap that person is backing up into a stove someone speak loudly and quickly... There is no evidence for you to believe that, but I hope you will, somehow, decide the intent was nothing more.


My opinion about ReactOS has nothing to do with my job or my employer, you are trying to make this into something it is not.


Your opinion has a huge amount to do with your job and your employer.

Appeal to authority may be a classical logical fallacy, but it's a highly effective one and is used by many people to infer the merit of an opinion rightly or wrongly so. Yeah, I know it's weird, that people would associate a comment one person posts with the voice of a trillion dollar company. Just happens sometimes.

Moreover I have nothing to gain by trying to "make this into" anything.

Believe it or not, I actually hoped it could be of some help to you as another data point - as far as all the things you take into consideration when choosing how to exercise your bill of rights free speech, which has nothing to do with your corporate free-speech. The latter of which unfortunately has cost many people dearly.

With respect to Microsoft, I'm certainly not making any comment about them as a company one way or another. That wasn't the point because, this concept applies to most large similar tech companies.

Things do change rapidly at such companies. Fwiw, I have done your job, your bosses job, been the janitor, and a few other things there. Things may be different now, but i'm not making the most possible uninformed guesses just to mess with you.

I sincerely hope the truth is known and that there are no negative repercussions for you personally.


I’m not speaking in the name of the company: I expressed my personal opinion about ReactOS and wrote back in 2017 that I think it’s a ripoff, after looking at their code. I’m not out there to destroy ReactOS, but if you ask me today what I think about it, I’ll give you the same answer.


Axel,

Look at my HN profile and contact me, so I can send you an email and ask you something. Or, put a way to contact you in your HN profile.


What is the point of comments like this besides aiming to seem smarter than the original poster? If you get your way then you would stifle open discussion and information.


I like this honesty and transparency, employers such as Microsoft encouraging it's employees to act in this way is a good thing.

Shutting down discussion behind lawyers and bland public statements should be discouraged.


We like it. But it can be actively harmful to the interests of the company. And its impossible for us to have the facts required to know if this is harmful. That's up to the company. So if they decide to fire employees who are spending their free time harming the company, we can't really get upset about that.

Its simply impossible to say, "no but this openness is good for the company" in an informed way. But we can certainly hope the company comes to this conclusion and encourages openness.


Collaborating across departments with people from different functional areas has absolutely nothing to do with dishonesty or lack of transparency.

If you have good people in those positions in your organization, they will come together and make good decisions.

If you don't have the right people in those positions, then that's not going to be a good thing irrespective of my advice.

Transparency doesn't require anarchy and anarchy doesn't buy transparency.


There are other jobs, would you really want to work some where that makes you afraid to have an opinion?




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: