You'd have to use absolute paths everywhere, but that probably doesn't hurt that much in programs or scripts. The CWD seems most useful in interactive shells, I guess. A fun thing is PowerShell on Windows where you have two CWDs, one from the process and another one from the shell which had is own VFS handling (e.g. the registry is a place that has no representation in the normal file system). Cmdlets use one of them and external commands and .NET APIs use the other. So in the latter case you always need Resolve-Path foo.bar instead of just foo.bar for things to work properly.
You wouldn't have to use absolute paths everywhere - you could use paths relative to any file descriptor that pointed at a directory. The shell itself wouldn't seem to have any problem - it could maintain a logical CWD without assistance. Utilities would probably need some other convention - maybe fd 4 points at the directory they are to operate in at start?
In a sense, this is "still a CWD" - but the differences would be 1) you can maintain multiple at the same time, and 2) you could close it.