Being a very experienced game developer who tried to switch to Linux, I have posted about this before (and gotten flamed heavily by reactionary Linux people).
The main reason is that debugging is terrible on Linux. gdb is just bad to use, and all these IDEs that try to interface with gdb to "improve" it do it badly (mainly because gdb itself is not good at being interfaced with). Someone needs to nuke this site from orbit and build a new debugger from scratch, and provide a library-style API that IDEs can use to inspect executables in rich and subtle ways.
Productivity is crucial. If the lack of a reasonable debugging environment costs me even 5% of my productivity, that is too much, because games take so much work to make. At the end of a project, I just don't have 5% effort left any more. It requires everything. (But the current Linux situation is way more than a 5% productivity drain. I don't know exactly what it is, but if I were to guess, I would say it is something like 20%.)
That said, Windows / Visual Studio is, itself, not particularly great. There are lots of problems, and if someone who really understood what large-program developers really care about were to step in and develop a new system on Linux, it could be really appealing. But the problem is that this is largely about (a) user experience, and (b) getting a large number of serious technical details bang-on correct, both of which are weak spots of the open-source community.
Secondary reasons are all the flakiness and instability of the operating system generally. Every time I try to install a popular, supposedly-stable Linux distribution (e.g. an Ubuntu long-term support distro), I have basic problems with wifi, or audio, or whatever. Audio on Linux is terrible (!!!!!!), but is very important for games. I need my network to work, always. etc, etc. On Windows these things are not a problem.
OpenGL / Direct3D used to be an issue, but now this is sort of a red herring, and I think the answers in the linked thread about graphics APIs are mostly a diversion. If you are doing a modern game engine and want to launch on Windows, Mac, iOS, and next-generation consoles, you are going to be implementing both Direct3D and OpenGL, most likely. So it wouldn't be too big a deal to develop primarily on an OpenGL-based platform, if that platform were conducive to game development in other ways.
I would be very happy to switch to an open-source operating system. I really dislike what Microsoft does, especially what they are doing now with Windows 8. But today, the cost of switching to Linux is too high. I have a lot of things to do with the number of years of life I have remaining, and I can't afford to cut 20% off the number of years in my life.
I'm a linux developer who's had an internship at a game company a
few years ago. I had the opposite experience.
During my internship, there were problems with visual studio.
Intelisense made everything so slow we had to disable it. VS was
pretty slow and was hard to use without being fullscreen on a big
monitor.
I use emacs and the gdb intergration is excellent. It's never
slowed me down. I've customized it and find it painful to use
another debugger (or an unconfigured gdb) so please don't have it
nuked. When I develop under windows (without cygwin), I miss many
tools I use when programming: valgrind, objdump, grep, sed. I
also miss having more control over the compiler and
linker. Makefiles seem to be much more powerful and flexible
although more complicated then most IDE's projects. SVN and git
are more complicated to use without the command line.
The game I worked on (deus ex human revolution) was being
developed for PC, xbox360 and ps3. Developping for ps3 means that
the code goes through gcc. This caused a few headaches for my
colleagues as it is more rigorous with standards (new line at end
of file gives a warning, templates need to have class or
typename, etc). I was used to it and actually like the fact that
it respects standards.
I've had to install windows 7 on a computer recently and was
baffled by the fact the realtek ethernet, USB3, video drivers
weren't included in the base windows 7 CD. Had I used the
manufacturer's recovery option it might have helped but my
computer would have been loaded with crapware. Ubuntu longterm
support works like a charm on that computer.
From what I've written so far you might conclude that I'm a FOSS
fanboy and dismiss everything windows has to offer. That's not
it, I'm used to my tools and everything else seems slightly
quirky to me. My tools aren't terrible and don't need to be
nuked.
Most game developers have had many years experience developing
for windows and use windows as their primary
platform. Programming for linux is different, strange and
possibly frustrating. I don't think the biggest issue with linux
game developement is the quality of the tools available under
linux but the fact that most game developers are used to using
tools only available for windows.
The one thing I think is lacking for linux to be a viable game
platform is quality graphics drivers but this seems to be
improving with Valve's recent efforts.
TLDR: Using windows is just as quirky as using linux. The biggest
difference is that you are used to your quirks as I am to
mine. Most game developers are used to windows and switching to
linux is hard and uncomfortable.
GCC doesn't "respect standards"; it has lots of own features and extensions that will make you stumble when actually writing code for a fully compliant compiler (not that there is one). You can still have makefiles on Windows; it uses the same toolchain setup internally that we are so used from the Unix world. And frankly, the debuggers on Windows are just completely.. ahead.
The Visual Studio debugger can step from managed code, into C++/CLI semi managed code, into native code. It automatically downloads symbols for all system APIs you are ever going to encounter, providing symbols for internal functions, too. It has expression evaluation on all these layers. Theres also the standalone WinDbg debugger that most importantly does kernel mode debugging but is designed similarly to gdb.
Also, of course - edit and continue. The way that GDB is decoupled from the compiler means that they probably won't ever be able to do that without major pains. Its a great design decision for when you need to debug unknown or hostile programs, but thats not the use case for most developers.
What I meant by "respect standards" was things like giving a warning if there is no new line at the end of a file, not accepting templates unless they are in the form "template <typename TYPE>" (the VS compiler lets you omit typename). I'm not sure which extensions you're talking about but every extension in GCC I know of is marked as such with a prefix. I think you'd find it hard to argue that the VS compiler is more standard compliant than GCC (or any other for that matter).
I know that you can have most of the GNU toolchain available under windows using cygwin. I use it quite a bit when I work under windows. I assume that the average windows game developer doesn't use these or he would find developing under linux very familiar (which hasn't been my experience while working).
The VS debugger features you mention seem to be compiler features rather than debugging features.
Now, if you're arguing over these details I'm afraid you missed my point: game developers aren't having a hard time developing games for linux because gdb doesn't support stepping from managed code into semi managed code. They are having a hard time because they aren't used to the tools available in linux. I wasn't saying the tools under windows are bad, rather that, as a linux developer, they can be as hard to use as my tools are to a windows developer.
I've spent the last seven years working on video game middleware and have had plenty of experience to many development environments. I've spent years developing for Windows, Linux, and Mac OS, and worked on every console that has been available on the market in that time.
GDB is different, but it's different in a bad way. Jonathan is completely correct in that it's a much worse experience because it doesn't easily give you the information you need to consume in order to find the problem you're looking to solve. Using the text user interface helps considerably but it still gets in the way. Sure, seeing source is fine, but what if you want disassembly too? Mixed source/assembly? Register view? Watches? Switching between threads easily? Viewing memory? DDD is an improvement but it still feels like it's 8 years behind. I'm glad to hear you're productive in that environment, most people (especially most game developers) aren't, and based on my experience getting people up to speed and fixing problems it's easier to get them going on Windows than on Linux.
That said, it's not the worst though. Green Hills takes that crown by a country mile.
The SN Systems debugger for PS3 is the best one I've used, bar none. It gives you all the data you need, is responsive, and has a few really nice features (write memory region to disk).
Not quite sure why you used gcc for PS3, SNC is far superior. It's faster at compiling and generates both faster and smaller code.
>Sure, seeing source is fine, but what if you want >disassembly too? Mixed source/assembly? Register view? >Watches? Switching between threads easily? Viewing memory?
Disassembly: (gdb) layout asm
Mixed source/assembly: (gdb) layout split
Register view: (gdb) layout regs
Watches: There is no window, which is too bad. You can say "display foo" and it will print the value of foo each time the execution breaks. You can also say just "display" and it will print the values of all variables you currently have set to display.
Switching between threads: Say "info threads" to show all the threads in your application. Say "thread x" to switch to thread number x.
Viewing memory: Again, there is no window, which is too bad. You can, however, use the "x" command, which is pretty full-featured and allows you to print the contents of a given-size chunk of memory, starting at a given point to the console in pretty much any format you might desire.
You may already be aware of these features, but be discounting for some reason?
Edit:
Turns out, GDB also supports writing memory regions to disk:
Can you share some of your more generally useful gdb settings / macros?
I've been working with linux/gdb for years and while I do agree it is powerful -- I don't find it very convenient.
For example, up until gdb 7.4 (pretty recent) there was no way to automatically skip certain functions -- the 'skip' command added to gdb 7.4 is almost a must for debugging heavy C++ code (and you still cannot add it to your .gdbrc file...)
Thanks for giving your insight, I found this post really interesting. Without going too far off topic, I was just curious, what kind of work did they have you do there? It just seems like the game industry doesn't get a lot of interns due to the complexity, cost, and time constraints of making major game like Deus Ex Human Revolution.
As an intern I was basically treated as a junior programmer. I had a senior programmer mentor me and usually ended up doing little, usually week long, tasks for him. He was a great guy and really made this enjoyable.
His main task was the cover system. I helped him with this as an introduction to the game engine. Once I had a grasp of the existing code and demonstrated that I was competent, he offloaded some of his simpler tasks on me. This usually involved modifying the game editor for the artists.
The modification I remember most clearly was enabling the artists to control the camera's FOV and the depth of field effect. The game engine didn't support this so I had to use a hidden bone in the models to control these values.
Working at Eidos Montreal as an intern was a great experience. It enabled me to work in video games (childhood dream) before realizing that it wasn't quite for me (I'm now trying my hand at entrepreneurship, it's all the rage).
> Someone needs to nuke this site from orbit and build a new debugger from scratch, and provide an library-style API that IDEs can use to inspect executables in rich and subtle ways.
LLDB[1] can do this, but I'm not sure how far along it is.
I use LLDB with some regularity. The last I checked, lldb is essentially at feature parity with gdb in user space (not sure about kernel debugging), and has additional features beyond that aimed at fixing some of the mistakes of gdb.
As a small example of something LLDB got right, say I want to put a breakpoint in this C++ template function:
template<int X>
factorial() { return X * factorial<X-1>; }
Because it is a template, this one line in code maps to multiple program locations (factorial<1>, factorial<2>...) gdb will create a separate breakpoint for each location, and each must be managed separately. But lldb maintains a logical / physical breakpoint separation: one logical breakpoint maps to multiple breakpoint locations, so you can manage all of them as a single value, and more easily treat factorial as if were a single function.
(Maybe more recent gdbs do this too - my version is rather old.)
One downside of lldb is that, while its command line syntax is very regular, it's also quite verbose. gdb's 'attach myproc' becomes `process attach --name myproc`. gdb's `up` becomes `frame select --relative=1`.
lldb does have a set of macros and abbreviations that cut down on some of the verbosity, but they haven't always worked well for me, and I hope they overhaul the syntax.
Judging by the version that's in Xcode, lldb is not ready yet.
The command syntax is a bit inconvenient compared to gdb. Perhaps I should overlook that, because it's presumably designed to be the lowest level of the debugger, so you interact with a GUI or something rather than with lldb directly, but Xcode's expression window is such utter unreliable junk that I end up having to use lldb by hand all the time anyway.
I've had problems with variable substitution. You can (say) ask it the value of $rip to find out what the value of the RIP register is, but if you try to set a breakpoint at $rip, it doesn't know what $rip is.
I've had problems with lldb telling me that a struct is forward-declared, even though I'm trying to print a local value of that struct type.
Sometimes lldb just gets confused, as in this exchange, which I reproduce verbatim, just as it happened:
(lldb) p this
(const CustomTrafficStyle *) $2 = 0x200cf4e0
(lldb) p *this
error: invalid use of 'this' outside of a nonstatic member function
error: 1 errors parsing expression
lldb doesn't have gdb's @ operator for printing arrays; instead, there's this separate command for dumping memory. So instead of "p x[0]@100", you might do "me r -ff -c100 `x`" - which is obviously a big pain, because it's a lot more verbose. I don't even know offhand how you'd use an expression for the count argument either (more inconvenient syntax.) (Furthermore, I don't even believe the me command does exactly the same thing, because I don't think it prints structs, but it usually does the job just about well enough.)
Finally, and most tiresomely, lldb will sometimes just print nonsense. So you might end up with something like this, which is all reasonable enough: (made-up example representative output, not a copy and paste)
(lldb) p x
(int *) $2 = 0x1234578
(lldb) p i
(int) $3 = 100
(lldb) p x[100]
(int) $4 = 123
(lldb) p &x[100]
(int *) $5 = 0x12345998
But then...
(lldb) p x[i]
(int) $6 = 0
(lldb) p &x[i]
(int *) $7 = 0xc4231777
As you hint at, Xcode is a part of the problem here. The debugging parts of Xcode (ignoring the horrible things wrong with Xcode as a whole) have always been a pile of junk, even before they introduced lldb. This is complicated by the fact that in my experience Xcode was always built upon a quirky Apple fork of outdated third-party tools.
So, I guess what I'm saying is: Even if lldb isn't ready yet (it probably isn't), you shouldn't judge it based on your experience with Xcode. Being integrated with Xcode would make any piece of software look like junk even if Knuth wrote it.
It's not very well known outside of Redmond, but windbg (provided as part of "Debugging Tools for Windows") is an excellent debugger. Steep learning curve, as the commands are cryptic to the uninitiated, but once you get the hang of it it's great. Visual Studio's debugger is absolute crap by comparison.
Thanks Jonathan for telling it like it is (and enduring the flames). As much as I hate Windows and like Linux, I agree on all your points. (Audio sucks ass on Linux!)
gcc/gdb tools suck because Stallman wants it this way. See Chandler Carruth's talk "Clang: Defending C++ from Murphy's Million Monkeys". Apple wanted to make gcc/gdb more modular, but Stallman doesn't want gcc/gdb to be modular and allow better tools because that would allow those tools to be non-GPL. So Apple spearheaded LLVM to ditch gcc.
LLVM/LLDB might be the way to improvements, but you are also right that the skills in building these higher level tools are not the strength of the current open source community.
So I don't end with a completely negative post on Linux development, Valgrind rocks!
The only good thing about GDB is that it can execute almost arbitrary C code in the frames. Otherwise it suffers from the same problems as GCC in that people decided that being able to embed GDB had to be as complicated as possible and it shows.
GDB only presents the GDB-MI (machine interface) textual interface as an "API". LLDB actually ships a library that you can link against. When it comes to integration, LLDB is miles ahead of GDB.
Well, it wasn't deliberate.
GDB is just so old that embedded it was never a serious concern when it was built.
At some point, it was decided the sanest way to make it "embeddedable" would be the GDB/MI interface, rather than a library.
The cost of Microsoft being unfriendly to game devs on Windows 8 is likely to be the cause of other platforms getting better, NOT people running in droves to Win8.
On even days I really wish they'd get this. On odd days I hope they never figure it out.
Linux sound has randomly become much better for the last year, including usb headphones. Don't know if its randomly, or if pulseaudio has finally started get some order to thing.
There is however a different way to look at the issue of sound support. Problems in linux tend to only stay unsolved when too few people care about fixing it. The few attempts made like alsa and pulseadio has been constantly criticized to have bad designs. One could then truly wonder why we don't see proof of concept design that is superior.
Perhaps the random factor is which devices you have tried. I haven't had a problem with this for literally years, though I don't dispute that some people have problems, this is because of the state of the drivers for the specific device.
When Microsoft owns the vast majority of the desktop market forever, and vendors do not make it a priority to support Linux, it shouldn't be a surprise that new hardware is being sold which is not well supported on Linux.
But Microsoft won't own the entire desktop market forever. If anything Linux is going to dominate that on the low-mid-range when Android and Chrome OS are merged and Ubuntu on the high end for workstations and gamers.
The main reason is that debugging is terrible on Linux. gdb is just bad to use
I am a cross platform developer. I have spent years developing on Windows, Linux, and OSX. (Also on other OSs, but what ever.)
Debugging with gdb is very different from debugging with a GUI IDE like Visual Studio. And if you are coming form Visual Studio to gdb, then debugging with gdb might seem down right impossible.
However, if debugging with gdb is so terrible, that means that each and every UNIX and all the software which runs on them were developed with this almost impossible to use debugger.
And Linus and the entire Linux dev. community, who rolled out Git almost on a whim, are stuck with gdb and can't come up with anything better.
Can that be true? Or is gdb in fact a great debugger, but with a very steep learning curve?
In my not so humble opinion, gdb is much harder to intuitively pick up than Visual Studio. But if do take the time to learn it, it is just as productive.
Actually in my experience, it is slightly more productive, because it forces you to work slower and think more. This go slower and think more strategy has also made my Visual Studio debugging better. Although if I spend too much time in Visual Studio I can slip into a pattern of less thinking and more break point setting.
tl;dr:
It's good that you are a very experienced game developer, but being a single platform developer kind of sucks. Become the kind of developer who doesn't care what platform they get to develop on.
Also, re this: "but being a single platform developer kind of sucks. Become the kind of developer who doesn't care what platform they get to develop on".
I agree with this in principle. But in practice... I invite you to develop large and complicated projects (like non-small games) and see if you retain this opinion. I find that work environment matters, a lot.
The thing that's a little sad is that developing on Linux could be great, if only open source developers had a culture of working a little harder and taking their projects past the 'basically working' stage and toward hardened completion. When things are solid and just magically work without you having to figure out how to install them or what config file to tweak, it drastically reduces friction on your productivity. So there's a productivity multiplier that open source developers are not getting, thus making all their work harder; because hardly anyone works on the basic tools, everyone else's development is more painful and takes longer, but nobody realizes this because everyone is used to it.
If someone made a debugger that let you inspect/understand programs 2x as efficiently as gdb does (I think this is not even an ambitious goal), how much would that help every open source developer? How much more powerful and robust and effective would all open source software become? (At least all software written in the languages addressed by said debugger...)
loved the first answer. I still remember the opengl 2.0 debates.
what puzzles me is the little mention of the horrible state of video, and audio in linux. alsa was an attempt to "fix" the audio sub(eco)system. the broken audio is probably one of the reasons why theres still no proper audio apps in linux.
video? well if you have a recent nvidia card on a desktop system you can at least play games. but that's about it. kernel developers make sure the closed source blobs don't use too many gpl features. nvidia blames it on the kernel devs, the kernel devs blame it on nvidia.
everyone talks about how awesome the opensource intel drivers are, and even though they got a lot faster recently with all the valve feedback, they were an order of magnitude slower than the windows drivers(i don't know if that's still the case). and mesa? well, you judge yourself.
I completely agree on your whole developer ecosystem. To me that is the sole reason why mac osx desktop utility apps strived while linux' more or les stagnated(no data just feeling, and maybe that's a bit harsh, but compared to osx apps that's certainly the case).
I don't know how anecdotal that is, but I've heard about pro audio users downright loving JACK [0] (not an audio app, but they must be using this toolkit for something).
Personally I like PulseAudio which has worked just fine for quite some time here on Arch Linux, while it always seemed to be a bloody mess of an integration on Ubuntu.
Jonathan, thanks for the post. Did you do the Braid Linux version yourself by the way, or did you get some external help in order to produce it ? From there on, based on your sentiment, does this mean that your upcoming games will not support Linux, or that you will let someone else take care of it ?
I recently asked Jon about getting Braid on steam and he said it being listed as Linux compatible was an error. So I think someone at Humble Bundle did the port for him.
>Someone needs to nuke this site from orbit and build a new debugger from scratch, and provide an library-style API that IDEs can use to inspect executables in rich and subtle ways.
The main reason is that debugging is terrible on Linux. gdb is just bad to use, and all these IDEs that try to interface with gdb to "improve" it do it badly (mainly because gdb itself is not good at being interfaced with). Someone needs to nuke this site from orbit and build a new debugger from scratch, and provide a library-style API that IDEs can use to inspect executables in rich and subtle ways.
Productivity is crucial. If the lack of a reasonable debugging environment costs me even 5% of my productivity, that is too much, because games take so much work to make. At the end of a project, I just don't have 5% effort left any more. It requires everything. (But the current Linux situation is way more than a 5% productivity drain. I don't know exactly what it is, but if I were to guess, I would say it is something like 20%.)
That said, Windows / Visual Studio is, itself, not particularly great. There are lots of problems, and if someone who really understood what large-program developers really care about were to step in and develop a new system on Linux, it could be really appealing. But the problem is that this is largely about (a) user experience, and (b) getting a large number of serious technical details bang-on correct, both of which are weak spots of the open-source community.
Secondary reasons are all the flakiness and instability of the operating system generally. Every time I try to install a popular, supposedly-stable Linux distribution (e.g. an Ubuntu long-term support distro), I have basic problems with wifi, or audio, or whatever. Audio on Linux is terrible (!!!!!!), but is very important for games. I need my network to work, always. etc, etc. On Windows these things are not a problem.
OpenGL / Direct3D used to be an issue, but now this is sort of a red herring, and I think the answers in the linked thread about graphics APIs are mostly a diversion. If you are doing a modern game engine and want to launch on Windows, Mac, iOS, and next-generation consoles, you are going to be implementing both Direct3D and OpenGL, most likely. So it wouldn't be too big a deal to develop primarily on an OpenGL-based platform, if that platform were conducive to game development in other ways.
I would be very happy to switch to an open-source operating system. I really dislike what Microsoft does, especially what they are doing now with Windows 8. But today, the cost of switching to Linux is too high. I have a lot of things to do with the number of years of life I have remaining, and I can't afford to cut 20% off the number of years in my life.