There are some things on the list that I totally get. We probably don't need support for .aif files in Python's stdlib, or support for the Sun directory service.
But I hate to see these get removed:
- cgi
- pipes
- uu
- telnetlib
A lot of people still use CGI. Yes, it's slow / low-performance / etc. It's also a perfectly fine way to make a knockabout internal service.
Pipes is a small pure-Python module, with nothing tricky about it. Why delete it? It's good at what it does. Subprocess equivalents are a lot more effort to write, even if you can eventually get to the same result.
Uu is old, yes. But it's not broken either. Folks do still uuencode things from time to time. Why delete it?
Telnetlib is personally useful to me, since telnet is one of the easiest ways to interact with a qemu VM's console. I hate that the answer is "Use something on Pypi instead", because that increases my dependency surface and adds no value for me. Why delete working, stable code?
The cgi module didn't actually contain the Python standard library's CGI support. That's in wsgiref.handlers.CGIHandler, and isn't going away.
What the cgi module did contain was some occasionally-useful support for working with HTML forms. As far as I can tell they're throwing that away for no good reason.
You're right, nothing is stopping anybody from putting these modules on Pypi. If somebody's already using Pypi dependencies, then another is no big deal.
But there's a really big jump from "This tool is self-contained" to "This tool needs X from pypi". It ripples into deployment, source-control, everything.
I try very hard to make tools that don't need anything except for Python's stdlib. That's based on a perception of trust: that the stdlib will be supported, and represents Python's only stable dependency.
Once the team starts to erode that trust, Python's compatibility story will start to break.
If you want to step up and be the maintainer of one or more of these modules in the standard library, I'm pretty sure you would be welcomed. PEP 594 is basically just a formalisation of the fact that no one has volunteered to do this yet.
Arguing that opponents to removal "can just step up as maintainers" are completely bogus. As mentioned elsewhere, this is more about eroding trust in stdlib as a stable baseline. This is just the kind of thing that will teach people to not trust Python as a stable foundation.
Also. If maintenance of aging modules becomes tiresome, maybe it is a strong hint to keep backwards compatibility in Python proper.
Not at all bogus. If "eroding trust in stdlib as a stable baseline" is an important issue to you, then you can do something about it by helping provide the resources that allows old modules to be kept.
Core language backwards compatibility is a non-issue. Approximately zero percent of maintenance costs go towards that.
Not to mention python has an entirely open proposal process where anyone at any point could have commented and expressed a desire to keep these libraries in the standard library. It seems after this entire process not enough people need them in the standard library.
If your implication here is that the very public backwards compatibility statement, the mailing lists where PEPs are discussed and approved/rejected, and the website python.org are a "locked filing cabinet", it may interest you to learn that this exact PEP was discussed over two threads and nearly 1000 posts on Hacker News: https://news.ycombinator.com/item?id=19948642, https://news.ycombinator.com/item?id=19985802.
That has no bearing at all on the PEP process and is a gross mischaracterization of it. It's clearly documented and open. If you subscribe to any python development mailing list or even the PEP github repo you would have seen this issue. You would have seen links to the discussion which was open for _three years_ and had over 100 comments on it. This was not a decision that was hidden in any way from the python userbase.
It wasn't a strong enough argument to overcome the final decision that cgi and cgitb are modules which never would be accepted in python today. There were three years to discuss this issue and make a case--was that process not enough?
cgi, nntp, telnet, etc. are old and obsolete. And not just in the 5 years out of fashion/not the trendy new thing--these things are decades out of date and have major or glaring security issues, etc. If you're publishing brand new code in 2022 that still depends on python's built-in cgi support, etc. you have far greater problems than having to pull in a third party dependency now.
Python changes in breaking ways--just look at the python 2 to 3 change. You will have to learn to deal with it. There is no explicit guarantee that python will continue to work in exactly the way it has always worked for you forever.
what's the major glaring security issue of cgi that you would consider a dealbreaker for a simple http-based service?
(this is an honest question, because when I saw this list I actually have a python script running on an apache web server that is using the cgi module to access post variables, and I never saw any need to change that, as any other "more modern" solution would be much more bothersome to configure and maintain.)
What is your auth and session story? Because I doubt you're implementing OIDC, JWT, etc. in a cgi script vs. relying on a modern framework that does all the work for you. HTTP basic auth? Well, there's your security issue right there.
So presumably you are happy to have a web accessible endpoint running on your server that will pop a new process and start executing a local script (which might be reading/writing files, other services, etc.) without checking any permissions, enforcing any cross site scripting or CSRF protections, etc. It might be fine for the most trivial appplication. For anything that actually changes state or has side effects, I would be very concerned and it would never pass a serious security review.
Firstly, HTTP Basic auth is not a problem over HTTPS (and it will be HTTPS more often than not, given modern browsers are bound to criminalise HTTP soon). Even then, putting an authenticating reverse proxy in front of such a site is dead-simple, and that can use wherever auth is required.
Secondly, chances are that if you're building a CGI site, you won't be exposing it to the outside world at all because it's internal/personal jank that's built to do a single job, and not to look nice. If it's meant to look nice and and handle the stress of being used by a public userbase, then it won't be CGI in the first place.
It's a massive footgun and no sane security review would let a production service pass with HTTP basic auth. You're one misconfigured TLS proxy away from major security breaches and issues.
* basic auth only exists at the proxy layer to protect apps with no auth.
* commonly deployed auth systems are secure without TLS?
because both are false. Check out the Rails Devise support for HTTP basic auth which is perfectly secure. And check that basically zero auth systems in the wild use PAKE and so are entirely dependent on TLS to secure the password transmission in flight.
How many "ands" should a theoretical security threat have before you should just ignore the guy proposing it?
E.g.
If mercury is in retrograde and the stars are aligned and it's Tuesday and the hacker is in position with Wireshark running on windows XP with a tethered pine phone, and we accidentally open all our ports, then we're completely vulnerable --> Ignore that guy.
I don't really see the connection between cgi and http basic auth. These seem orthogonal issues.
(And fwiw the usecase I have does not have any authentication and doesn't need any. It's taking a POST var and doing things with it, but there are no secrets involved.)
WebOb uses cgi for parsing multipart uploads/forms...
It's not obsolete, and it is heavily used, it's working software that works well. Ripping it out just means now there will be N copies floating around that will all need to get patched/fixed instead of just a single version included with Python.
Well apparently it's not heavily enough used that any of its current users were motivated enough to discuss the PEP and make a strong case for keeping it in the standard library. The discussion was open for _three years_ and had hundreds of comments: https://discuss.python.org/t/pep-594-removing-dead-batteries... The outcome is that there isn't enough usage to merit this being in the standard library.
I suspect there's very little overlap between users of development software and participants in the bureaucracy that designs the software. The requisite personality types are very different.
If you're developing software that depends on a python standard library it is on you to subscribe to the relevant mailing lists so you can be notified of security issues, deprecations, changes, etc. There is not bureaucracy here to be aware of what's happening, or even comment on the changes.
Years ago I used to do that. But nowadays I use so much external stuff it's impossible. So I'm actually immensely grateful to the Python team for including so much in the Standard Library and maintaining it for decades - because for me, at least until now, it's been that one part where I could relax a bit about and be confident that competent people will take care of everything and I won't need to spend much time analyzing the situation.
Seems to me that this is exactly the problem. The python team is saying they really can't maintain these libraries to a high standard and need to remove them from the standard library. I can see both sides but really that makes a lot of sense to me.
Is my comment. It's the 6th comment down. Clearly I didn't make a strong enough case, but I certainly did advocate for it and mention that it is in use.
Since NNTP was my gateway to open source, I'm somewhat sad to see Python's nntplib module die... but NNTP is quite well advanced in its way out. If you were asking a decade ago, NNTP would be a relatively niche community that potentially had some relevance, but posting volumes not associated with alt.binaries has declined precipitously in the past decade alone (judging from aioe.org's statistics, it's a third of in 2022 of what it was in 2012, and even that feels generous).
I can understand some concern about the impact of removal of support for deprecated things where it potentially impacts archival scenarios (e.g., uuencoded text), but dropping NNTP from the standard library is probably the right call for Python. It's not like the protocol is so complicated you need a library to do it for you, anyways. ;-) (To be fair, I often use NNTP as my example protocol for implementing stuff because of its simplicity.)
Telnet the protocol is secure if you either (a) use the Telnet ENCRYPTION option (such as with Kerberos ktelnet), or (b) run it over TLS (standard practice in IBM mainframe environments in which Telnet is still heavily used). Most Telnet clients and servers, this Python telnetlib included, never implemented either. But people present the protocol itself as being insecure, when it is just that the majority of implementations of it never bothered to implement the security extensions which have been defined for it. Telnet-over-TLS is just as secure as SSH. Telnet has the advantage of having various standardised extensions which don't exist in SSH, such as IBM 3270 and 5250 block mode terminal support, the SUPDUP protocol used by PDP-10s and Lisp Machines, serial port control (enabling you to configure baud rate/parity bits/etc if you Telnet to a serial port server), and Kermit. Now, you may not use any of those features – in which case SSH will suit you perfectly fine – but some people do. And, in principle someone could define equivalent extensions to SSH, but by and large nobody has done so, and even if somebody did, they wouldn't be standard so little or no software would support them.
It has three functions: one to encode, one to decode, and one to show usage information and handle arguments. It's only had 3 commits since 2010. There just isn't much to maintain.
Meh. CGI was one of the easier to consume interfaces that is really not hard to rebuild from scratch. I suppose it makes sense that python has (or had I guess) a library for it but it's really not necessary. Enough of the primitives are there. Those that need it could easily write a shim.
It does make you wonder why it was removed though. I can't imagine it was difficult to maintain.
I would express all of the concerns being expressed here were debated ad nauseam over the three year political process of getting this proposal accepted. If you want to vent your fears and frustrations in a hopefully more productive way, you can get involved in Python's process.
All that is to say, I'm thankful that the Python community gets things done, and people are willing to go through three year processes for changes such as this.
I also want to highlight these packages will be part of the stdlib for python 3.12, which will EOL in October 2028. In other words, if this change will affect you then you have up to 6.5 years to figure out and implement a plan.
I thought that feedback from the first discussion was carefully considered and I'm pretty happy with take 2. I have used the some of the modules being removed but largely agree now they are either legacy or there are significantly better third party options.
I don't understand the motivation to remove MSI support. The document says:
> Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as a new deployment model
First, MSI has better OS support than AppX, second, many users are still using Windows 7 and won't be able to use AppX. MSI is not some 30-years old deprecated standard.
FWIW, CPython officially follows Microsoft’s support schedule on Windows systems, so versions ≥ 3.9 won’t even install on Windows 7. (Some reasonable people like it and some don’t, but this has been the official stance for a long time.)
As for MSI, I’m torn: it is the most widely-supported and reliable packaging approach for Windows and, at the same time, a 20-year-old undocumented dumpster fire of an archive format spliced together with a scripting language forced into the mold of an allegedly-SQL relational database for which it is the sole user, and I can’t really find it in myself to blame Microsoft for wanting to kill that.
(At the same time, they also cheerfully promote an admittedly documented 10-year-old metadata format [WinMD] grafted onto a bytecode+metadata format for a different platform [.NET] packed into a Win32 executable format [PE] that is built upon an early and obsolete SysV object file format [COFF] with a stub executable for DOS [MZ] prepended in contravention of the original spec—as a substitute for a 30-year-old undocumented metadata format [TLB] that is actually two different incompatible formats [SLGT or MSFT] but at least doesn’t have so many layers, so I can’t exactly promote them as a shining beacon of logic and sanity either.)
Windows 7 is the best version of Windows Microsoft ever made, and I won't give it up until I have absolutely no choice. I'm already disappointed that I can't get Python 3.9 - I don't understand why they have to actively bar you from installing it rather than just telling you to use it at your own risk.
Because it prevents whole classes of support issues - if you can't install it, you can't raise a ticket complaining that something doesn't work under 7 and then waste support time... even if you say "if you're running windows 7 you're on your own" people will still try it on, or ask for one little tweak to make things work... all effort that could be better spent on forward progress or real support issues. It is far easier to prevent people from being able to have issues in the first place.
Also, I agree that 7 was a fine OS, but it is more than two years in the grave... you cannot expect continued support for it. It is sad, but a statement of fact.
While that's true, you also aren't getting security updates - there have been 222 CVE score 7.0 or higher vulnerabilities for Windows 7 (all versions) assigned on or after the EOL date of 2020/01/14, some of which are rather nasty. While a couple of the most extreme ones did receive patches, the vast majority have not and a fully up to date Win7 system will be hugely vulnerable.
Given that any form of internet connectivity exposes you to a highly adversarial environment, I would personally be very uncomfortable persisting with Win7 - all it takes is one drive-by browser vulnerability and the attacker has a whole host of working exploits that can be used to further compromise your workstation.
I will not bother you further with the security aspect, but circling back to Python - your operating system does not exist in a vacuum, compatibility with Win 7 is actively degrading as it remains the same while Python and everything else continues to change, so it makes sense for them to prevent it being installed. That way the support community's time is not wasted even reading enough of a support issue to see that's on Win 7 and closing the issue - one or two in isolation would be tolerable, but in bulk even assigning or closing tickets would take a decent chunk of resources.
It's interesting because once Win 10 support drops there will no longer be any x86 Windows path for Python, but there's still a lot of x86 Windows hardware that still works.
I suppose that's more an argument to move to Linux/FreeBSD though on that hardware... I can understand dropping support for something but single breaking changes shouldn't merit it.
This PEP doesn't really stop functionality for nearly 3 years and even then it's only with the newest Python version so people can stay on the older version for another 2-3 years of further transition fully supported by the Python community.
In 5 years, Windows 10 will be the bare minimum for a secure OS from MS.
Why can't legacy MSI support be a standalone library? It looks like no one is maintaining it right now anyways so that should be some red flags right there.
The functionality in the msilib module is somewhat low-level anyway (it basically just opens the database for you and leaves the arcane incantations required for doing anything as an exercise for the reader) so it probably wouldn't be hugely difficult to replace it with ctypes calling msi.dll directly.
If I were to replace my msilib-based installer, I would migrate to Wix# [1], a fantastic high-level wrapper for WiX that lets you express your installer in a few lines of C# (and then generates the thousands of lines of XML that WiX needs, but you don't need to touch it). I wouldn't have used msilib if I had known about Wix# at the time.
I don't have any data on it, but I'd speculate that CGI is still fairly widely used. Though how widely it's used with Python is another issue. Regardless of how poorly designed the cgi module is, removing it without providing a one-module migration path in PyPI doesn't seem like an obvious win.
> For example, least controversial are 30-year-old multimedia formats like the sunau audio format, which was used on SPARC and NeXT workstations in the late 1980s
Why would something like that add maintenance burden to the Python devs? I'd expect such code to be very, very stable.
Many software projects impose targets on themselves for things like 'software quality' that involves even stable code being checked and changed.
For example, maybe they've received reports the code has bugs, but nobody thinks fixing is a good use of maintainers' time - or with such an old format, maybe no-one even knows what the correct behaviour should be.
Maybe they have decided all file loading and network-interaction code should be fuzz-tested for security bugs. Or that every such module should have at least x% unit test coverage. Perhaps the the code uses deprecated methods and needs to be modernised. Or maybe they simply feel every file ought to have an owner, and nobody trusted has stepped up.
It's good to see this happen. One concern I have with Python is that its standard library is large enough to make reimplementation impractical. It even requires another language, TCL, to be included. Python could be bound to CPython forever. Removing the 'dead batteries' should help somewhat with this problem.
As someone who has simple Python code for a CGI service that runs a couple of time a day (Golang vanity URL thing), that's disappointing. Maybe I'll rewrite it in Go to improve performance even more.
They mention that one reason to remove these modules is storage savings, but then there is no information about how much storage will be freed up by their removal... Anyone have an idea if it is significant?
This excludes the "msilib" and "nis" packages, which aren't on my system (I guess my distro excludes them?), and you should also really round up the files to the disk block size (usually 4k), so let's round it up to 1M.
Is that significant? You decide. It doesn't seem all that significant to me given that the stdlib is 41M, so for space-constrained use-cases you're still not going to be able to copy/install all of the stdlib in the first place as it's still not even close to fitting on "devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit)" that the PEP mentions.
I do think that removing these packages is a good idea, but "space savings" seems like a very weak argument.
> you should also really round up the files to the disk block size (usually 4k), so let's round it up to 1M.
A decent first-go model for the amount of bytes to add is half a block per file. You’re adding about 168kB. That model would predict that for 84 files. I guess there are quite a few more in those (at least) 18 directories.
Oh, you're right. I was thinking NDIS for some reason. Probably the distribution does not support NIS and/or does not enable it in their Python builds.
>> Internet is full of entitled narcissistic people. They can't imagine there're other important things in the world beside themselves and their needs.
I completely agree with you, but I'm not sure which side of the dead batteries debate you're supporting with it! ;-)
I'm genuinely curious to see how this goes as a lot of languages have the same dead-battery problem.
In Java for instance, StringBuffer and StringBuilder offer a very similar interface but one offers locking. Nowadays, most locking concerns are left to the caller, not the called. HashTable/HashMap are another example... The nice thing is, old libraries "just work". Java is trying to hide older parts of the API with Modules, and it remains to be seen whether or not this solution will take off and it will be adopted into other languages.
> A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit).
It probably would, but the standard library is never going to be reduced to that size (I hope!), so how do these removals benefit those devices?
If they're deprecating part of the stdlib where a third-party lib is favored (e.g. 'lxml' instead of 'xml', per the article), will they be bringing the third-party module into the core stdlib?
Surprised the ones without a replacement don't have PyPI modules available. I'm sure some enterprising fellow will simply extract the code and switch it into a PyPI module.
I feel that every release, the python team should do some soul searching and pick 5 modules to downgrade to external package(pypi) status, then pick 5 deserving external packages and elevate them to base status.
This would, for one, add a bit of churn to the standard library leading to a more healthy ecosystem. And two, give everyone a target to complain about, I mean, people are going to complain about something, might as well focus their attention.
Very few modules in the standard library actually get deprecated, that's why this PEP got put together. It also took 3 years of discussion to achieve consensus and the initial list was whittled down quite a bit and the reasoning became much stronger on the final list. And these aren't even being removed until Python 3.13!
The SC have also said this is a one time event and they won't accept a similar PEP in the future, modules in the future most be handled on a case by case basis and the preferred outcome will not be to remove them.
That would result in a mini-Python 2/3 catastrophe every release. No, thank you. Maybe that's acceptable when you're the author of a piece of software, but as an ops engineer, that prospect makes my hair stand on end.
Because dependency hell isn't dreadful enough? Some entries will be completely missing from requirements.txt because the person assumes a certain version of python is present.
> CGI is deemed as inefficient because every incoming request is handled in a new process.
This is why I keep every program I use on my laptop running in the background at all times. Imagine having to wait for a process to start before you can use it.
"Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance cost frees development time for other improvements." This really doesn't jibe with the "Hey guys we have tools to convert 2 to 3 so easily" and "we aren't breaking stuff." A good working library will continue to be a good working library until you break it.
"Modules in the standard library are generally favored and seen as the de facto solution for a problem. A majority of users only pick third-party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of xml. The removal of an unmaintained stdlib module increases the chances of a community-contributed module to become widely used."
First, part of the appeal of Python is that de facto standard. Recall all of the XKCD comics about it. I don't have to make choices about which of seventeen different Perl libraries I can maybe find that exist on CPAN and try to decide between them. This is one of the reasons why I fled Perl.
Second, why do we want a community-contributed module to be widely used? That's stated as a goal but without explanation. If there is a widely used community-contributed module that is better, put it in the standard library.
I could almost like "A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage ..." except that there ought to be a mechanism for cutting parts out of the standard library anyway if they aren't being used by your particular space-starved platform. This doesn't actually solve the problem being posed. It isn't the sunau module that is making or breaking the memory budget -- find a way to dis-include unused stdlib modules for projects.
Listen to this: "The crypt module has fundamental flaws that are better solved outside the standard library." So either fix them or move whatever is being used into the standard library.
Now, we can quibble about individual modules (I am rather fond of cgi, it's just so damned handy if you want to make something small without hauling in entire frameworks), but that misses the point -- the standard library exists to free us from the burden of sifting through N implementations of whatever it is we want to solve, where the complete coverage of the issue varies from "the happy path" to "eighty percent of the problem but sadly you need to do work in the twenty percent not covered."
> Second, why do we want a community-contributed module to be widely used? That's stated as a goal but without explanation. If there is a widely used community-contributed module that is better, put it in the standard library.
I think the reasoning is that offering a flawed version in the standard library can prevent an alternative from being written and gaining adoption.
I think that's it. If you haven't heard of Requests, you might re-implement it yourself from the standard library. That's totally possible (and I had to do that back in the dark days before Requests and pip), but not something most people should be doing.
The tl;dr is that if you move a package into the standard library then you are restricted by the python feature release schedule (was every 18 months, now annual) and backwards compatibility requirements (see https://peps.python.org/pep-0387/).
> A good working library will continue to be a good working library until you break it.
This isn't necessarily true, it depends on what the library is targeting. Sure, you could try to use a libcurl 7.9 from 2001 but there's going to be security bugs abound and it's going to be a pain to send verbs like PUT that some APIs expect. Standards, security concerns and paradigms change over time and can easily break good working libraries that are unmaintained.
However, things like a json parser and writer are relatively stable and don't need many changes over time because of the stability of the format.
But I hate to see these get removed:
A lot of people still use CGI. Yes, it's slow / low-performance / etc. It's also a perfectly fine way to make a knockabout internal service.Pipes is a small pure-Python module, with nothing tricky about it. Why delete it? It's good at what it does. Subprocess equivalents are a lot more effort to write, even if you can eventually get to the same result.
Uu is old, yes. But it's not broken either. Folks do still uuencode things from time to time. Why delete it?
Telnetlib is personally useful to me, since telnet is one of the easiest ways to interact with a qemu VM's console. I hate that the answer is "Use something on Pypi instead", because that increases my dependency surface and adds no value for me. Why delete working, stable code?