RE: effective_io_concurrency --- any idea how this should be tuned for SSD or PCIe flash? The manual only gives an example of how to tune for RAID (by number of drives).
From my experience, for each SSD drive you can set it between 5-8 (if there are in raid0).
BTW in 9.6 the effective_io_concurrency  can be different for each table space (especially useful if you have multiple drives of a varying quality [ssd, spinning] in one server).
On modern SSD, values 5-8 per device are definitely on the low side. See for example this thread from 2014, on (then released S3500 SSD from Intel):
That shows the performance increases up to effective_io_concurrency=256 (of course, in reality there may be multiple queries running on the system, sharing the I/O paths).
I did some testing a while ago and didn't see much gain past 30 on 4 SSD disks. Should do another round of testing. Thank you for sharing.
I'm curious as to what the performance looks like when it's properly tuned.