Most likely the fact that they’re running ancient kernel versions (RHEl 7.6, released Oct 30 2018 uses Linux 3.10, which was released in June 2013) , and have to support them for 10-20 years.
RHEL 6 uses kernel version 2.6.32, released in 2011, and will be supported until 2020.
With a moving target like Btrfs, which may have plenty of showstopper bugs left in it yet, it’s just not feasible to keep backporting these new features to a an old kernel version.
Once Btrfs stops getting major patches, and is considered stable, I guess it could eventually make its way back.
While true, there's also the fact that they don't have any btrfs engineers anymore (from what I've heard). The ones they had have all left for Facebook and other companies that use btrfs in production.
Redhat have added new filesystem support after a .0 release. For example, they added XFS to the kernel (without xfsprogs) in RHEL 5.2 and added the tools to the base repo in 5.4, then officially supported it in 5.6.
XFS is a conceptually simple journaling file system, comparing that to the crazy corner cases that can exist within btrfs snapshots and volumes and subvolumes and redundancy is not reasonable. A few years back btrfs had a lot of momentum, but other projects have matured since then and it just doesn't make as much sense as it used to. btrfs was meant to be ZFS without the license problems, but ZoL has gotten significantly better, and for some legal reasons that are fuzzy, Canonical now supports and ships ZoL in Ubuntu. lvm/dmraid performance has gotten better and integration with LVM and other software has improved, to the point that having a standardized interface for managing your storage (Like ZFS and BTRFS have) is no longer such a big deal.
Between lvm/dmraid and how much effort they've done to make it simple to use lvm storage for libvirt/openstack/etc., I see no reason why they'd ever support btrfs in RHEL. They would spend more in engineers than they would ever get back in support.
You are probably right in terms of dev and support overhead. I would imagine that should they decide to add it, the process would be similar to XFS; in that, they would add support for it in the kernel and call it "experimental" like they did with XFS in 5.2.
XFS is a fairly simple filesystem. That said, the issues they had were integrating it in VFS. There are still some bugs at that layer today, albeit minor ones. In 5.x, most of the issues were race conditions in VFS and raid controllers (sas bus resets and such). XFS itself is actually very mature but any time they add a new FS, the VFS layer requires a lot of work.
RHEL 6 uses kernel version 2.6.32, released in 2011, and will be supported until 2020.
With a moving target like Btrfs, which may have plenty of showstopper bugs left in it yet, it’s just not feasible to keep backporting these new features to a an old kernel version.
Once Btrfs stops getting major patches, and is considered stable, I guess it could eventually make its way back.