
RaptorCS Power9 Blackbird PC: An expensive mistake - stargrave
https://drewdevault.com/2019/09/23/RaptorCS-Blackbird-a-horror-story.html
======
kev009
I lost a Talos II board due to what we are fairly sure was a shitty PSU on the
first few powerups. I replaced the PSU with a store bought Corsair one and the
new unit and replaced (under warranty) board has been very reliable.

Raptor is a small company and one guy is fronting most of the support load so
keep trying, he's not out to screw any customers or provide an inferior
product.

~~~
ksec
>Raptor is a small company and one guy is fronting most of the support load so
keep trying, he's not out to screw any customers or provide an inferior
product.

But Radio Silence for a few months... surely that is a bit too much?

On the other hand I didn't know Raptor was this small.

~~~
tpearson-raptor
It's not that small, the information above is not accurate.

However, I think people that do not see what we see in the support system are
greatly underestimating the volume of Email an OEM/ODM receives on a daily
basis, which then must be filtered and sorted through to determine the valid
issues from the flood of spam and general inquiries. It's not something we
have a solution for yet.

------
Porthos9K
That's a shame, but I hope this doesn't deter others from trying and
popularizing non-Intel architectures. We need to break their grip on the
desktop/server/laptop market.

~~~
prongletown
>We need to break their grip on the desktop/server/laptop market.

I'm curious as to why you think this "needs" to happen?

I have my own reasons, for example, openness of ISA and IP from up top at the
browser all the way down to the firmware on the chip; however, I'm curious to
hear yours.

~~~
Porthos9K
> I'm curious as to why you think this "needs" to happen?

I think Intel has been too dominant for too long. They've gotten complacent
and sloppy, as shown by microcode-level vulnerabilities like Spectre and
Meltdown.

Also, I figure breaking Intel's dominance will further marginalize Microsoft.
I still hold a grudge over how horrible an experience Windows 98 was.

~~~
iforgotpassword
Huh, 98SE was the sweet spot in the 9x series.

Fwiw, the NT kernel was designed for portability from the ground up. During
initial development they targeted x86 and the i680 and it has been ported to
pretty much any CPU that had relevance over time. Itanium, Alpha, ARM and....
PowerPC. While that branch is probably not maintained currently it would be
pretty easy for Microsoft to get that going again.

~~~
Porthos9K
I'm still hoping they'll do the smart thing and fork FreeBSD. Why EEE Linux
when there's a perfectly good Unix with a business-friendly license?

~~~
IntelMiner
Because they don't have to EEE Linux

If you want to run Linux in the cloud, you've got a bunch of providers
(Google, Amazon, Microsoft, etc)

Microsoft simply wants to be in that market, so they'll offer the same
products their competition does. Anything they can do to make Linux better on
Azure is considered a net gain for them

------
glitchc
Sounds like a faulty PSU. $70 USD is quite on the low end of the spectrum,
especially for a chip with a TDP of 130W. Is it a working pull unit? Do you
have the ability/option to try another unit?

~~~
classichasclass
The chip he appears to have bought is a 90W TDP part. I have a pretty beefy
650W power supply in my Blackbird, but I think that was overkill, and it was
$94. I would certainly try another PSU but I wouldn't immediately think that
would be the problem.

------
eatonphil
This is really disappointing to hear. I've been thinking of getting one.
Thanks for documenting the issues.

------
lightedman
Symptoms pretty much guarantee bad or improper RAM.

Source: I've been building systems, including POWER systems, for 30 years.

~~~
tpearson-raptor
You're pretty close, though in this case it's apparently the RAM controller
inside the CPU itself. This failure mode is incredibly rare but it just
happened to hit this one system, unfortunately.

------
nullc
I have three talos II boards, no issues with them except I stupidly cut power
to one during a firmware update.

The raptor co staff was exceptionally helpful to me on IRC in fixing it.

------
CharlesColeman
How many employees does RaptorCS have? Their products strike me as very
ambitious but quite niche.

~~~
tpearson-raptor
This information is not public, however I do want to note that the products
are not exactly niche. We'll have some announcements shortly that will help
clarify where we intend to go with the newly created POWER desktop market
segment.

------
kkarakk
As much as I like the whole "oh we're a boutique business!" model this kinda
stuff has been my experience routinely with anything computer related. Don't
be the small business/entrepreneur outlier unless you're a multi-million
dollar business, they can't afford to care about your 2k dollars too much

------
gautamcgoel
I appreciate Drew sharing this, I was interested in picking a Raptor board up
sometime...

------
gautamcgoel
Update:
[https://twitter.com/RaptorCompSys/status/1176432946670186498](https://twitter.com/RaptorCompSys/status/1176432946670186498)

------
grubles
Please don't dismiss Raptor over this. I've had nothing but a great experience
ordering from them. I do not have a Blackbird but my Talos II Entry system
(1-cpu slot vs. 2) works flawlessly.

~~~
iforgotpassword
So you were lucky to receive a system that just works without issues so you
never had to contact their support in the first place. That way of reasoning
would work for even the shittiest company on this planet.

I'm really glad I read this piece. The geek in me was playing with the idea of
getting a Talos 2, but the pricing makes me really hesitate. Knowing that if
I'd end up with a problematic unit their support will be absolutely worthless
is important information.

~~~
tpearson-raptor
We've shifted our policy to allow rapid RMA of problematic units, which is
more in line with other vendors. Please be assured you won't be stuck a
problematic unit, especially with our well-established Talos II line. For
Talos II, which has been on the market for two years, we have a well
established support system in place, which asks for some basic diagnostics to
verify a faulty board (this is rare) before authorizing RMA for repair or
replacement.

With Blackbird there was more of a focus on trying to understand any new
faults found in the field, given their tendency to disappear when the units
were brought back for analysis. Over the months after Blackbird shipments
started, we learned that the majority of those were related to improper
customer component installation (CPU, RAM, PSU, chassis), and are in the
process of expanding the existing RMA policy to cover the Blackbird line.

------
tpearson-raptor
Timothy Pearson from Raptor here. I'd like to first extend apologies from the
apparently poor support experience from Raptor Computing Systems -- that being
said, there are a few exceptional circumstances in play with this system:

1.) This particular case is one that Raptor needed to consult with IBM
engineering to determine the correct course of action for. There is a very
rare CPU failure mode (we have only seen it happen twice) that manifests with
similar symptoms and a ZCAL failure. The root cause here is an apparently
defective CPU that slipped past IBM final test, and work is underway to make
sure that these marginal, but defective, CPUs are caught instead of being
shipped out.

2.) There is definitely more than one person in support, however Raptor has
been overloaded with both spam and off topic inquiries. Raptor has attempted
to manage this load without much success, despite deploying new features such
as the troubleshooting guide at
[https://wiki.raptorcs.com/wiki/Troubleshooting/Support_Reque...](https://wiki.raptorcs.com/wiki/Troubleshooting/Support_Request_Checklist)
we continue to receive support requests that are missing vital information,
and with descriptions that are so generic that the problems could range from
"PC not plugged in" to "used non-ECC unbuffered RAM" to "tried to install
Windows" (yes, all of those have happened, repeatedly).

3.) We are currently gearing up for a major push on owner-controlled
computing. Higher level technical resources are assigned to those projects
temporarily, which means unusual issues like the ZCAL problem are being
resolved at a slower rate than normal.

I'd ask that the general community please bear with Raptor as we work through
these problems. What we're doing is _hard_ , it's not as simple as shipping a
RISC-V SBC in a known configuration, we have the entire complexity of the
traditional PC ecosystem in play. AMD's release of the uber-fast (but ultra-
locked-down and owner-disrespecting) Rome CPUs has not helped the owner
controlled computing world, and I am still of the opinion that POWER is a
better long-term investment than RISC-V in the desktop and server space. FWIW
I use a Blackbird desktop as my daily driver, and have not had any complaints
with it (in fact it's given me less problems over time than my previous daily
driver -- a corebooted ASUS KCMA-D8).

Raptor has a Wiki with a ton of information available at wiki.raptorcs.com and
I am definitely open to further suggestions on how to make the nascent POWER
desktop community a bit more self sufficient than it is now. Remember that x86
vendors don't need to offer significant post-sale support, partly because the
Linux community has already filled that role, and as a result they can
continue to offer low prices, strengthening the lock-in effect. If owner-
controlled computing is to succeed in the marketplace, we need to figure out
the best way to ensure that vendors aren't bearing the brunt of those costs
and being forced pass them on in higher product pricing.

I believe support has reached out to you now that we understand the CPU will
need to be replaced. Can you confirm this?

Thanks!

~~~
Sir_Cmpwn
Hi Tim, author here.

>I believe support has reached out to you now that we understand the CPU will
need to be replaced. Can you confirm this?

Yes, support reached out. I said as much in the email, but to address some of
your comments publically:

(1) Totally understand this and I posess a tremendous deal of tolerance for
weird problems with unusual hardware from small companies. But the support
experience here was abysmal and no amount of weirdness in a problem excuses
the lack of communication. I would be with you every step of the way if Raptor
had communicated the progress - the first I heard that IBM was even being
consulted about the issue was today - or even sent me communications at the
times they promised to in earlier replies.

(3) I think you had really better fix (2) before you increase your volume. The
reply I received from support today insinuates that my support experience is
rare, but I can't throw a stone in the IRC channel without hitting 3-4 people
who have had similarly bad support experiences. Whatever investment you're
putting into your upcoming "major push", you should probably siphon off some
of it for investing in better support.

It also bears mentioning that 50+ days of silence, not including an additional
30 days where communications were devoid of information outside of unfulfiled
promises about future communications, followed by a detailed response on the
day I publish a scathing review online, is worth note. I should hope that
other customers who don't have a soapbox to complain on are also having their
concerns addressed.

~~~
tpearson-raptor
This was originally going to be published on the internal ticket, but since
the issues were raised here I'm going to post here instead.

First, I'd like to point out that Raptor already identified one of the major
problems here -- that front line support failed to keep you notified of back
end progress in diagnosing the problem. Raptor Computing Systems is an OEM and
ODM first and foremost, not a services and support firm -- their capacity to
handle requests is limited by the amount their customers are willing to pay at
time of product purchase, and internal management of RCS is trying to balance
support costs with all other costs and overall product pricing. They may not
always get it right, but they are trying nonetheless.

Yes, this was brought to my attention as a result of the review. That review
has apparently caused sufficient damage that Raptor will now be reevaluating
whether they can afford to service the individual market at this time, or
whether they need to focus on business users only.

My personal take:

I am saddened to see x86 and locked computing profiting from this, but at the
end of the day individuals have always been a difficult segment to target.
They tend to want extremely low (below cost) pricing, high levels of service,
and generally want to run various proprietary games on Windows while
simultaneously settling for a false sense of security through open source
firmware wrapping large hidden binary components. Out of sight, out of mind,
purchase price above all other concerns seems to be the general takeaway for
that market.

As a result of this mentality, individuals are not currently Raptor's core
market, and at the end of the day the individual computing market will get
what it is willing to pay for. If that's locked (but cheap) x86 computing,
open source itself may be in serious trouble due to the impending final
lockdown of those platforms. If it's instead cheap low power computing
(RISC-V, low end ARM) there's an awful lot of fat that will need to be removed
from near-essential projects like Qt, KDE, Gnome, Firefox, Chromium, GCC,
Clang, etc. to better fit them on the lower end platforms. I have not seen any
developers yet willing to put in the time and effort to re-architect those
systems for low end computing resources; in many cases the fundamental design
would need to be changed (i.e. remove interpreted scripts as much as possible,
go back to C/C++ or a similar low level language, etc.). In the case of
projects like Chromium, it may simply not happen as the intent of that
particular project is partly to serve ads and DRM protected content through a
portal system, which aligns nicely with the idea of locked end user computing
on x86.

~~~
Sir_Cmpwn
This reads like blame shifting, not accountability.

> Yes, this was brought to my attention as a result of the review. That review
> has apparently caused sufficient damage that Raptor will now be reevaluating
> whether we can afford to service the individual market at this time, or
> whether they need to focus on business users only.

See: the review caused the damage, not Raptor's failure to provide a good
support experience. In any case, if you can't handle the support load, then
you're probably right in reconsidering.

I am really rooting for Raptor here, I'm not your enemy. I agree that x86
monoculture is awful for the ecosystem and ARM hardly presents anything
better. This is also why I got heavily involved in RISC-V, and I wanted to
find something similar in POWER9. I haven't laid hands on a working Blackbird
yet but if it is as good as I hope it is, then I think you have a good shot if
you just focus on scaling and fixing your support problems. If you can solve
the support and get the system working, I would be thrilled to amend my review
and publish a new one with all of the things I like about my shiny new
Blackbird system.

> I am saddened to see x86 and locked computing profiting from this, but at
> the end of the day individuals have always been a difficult segment to
> target. They tend to want extremely low (below cost) pricing, high levels of
> service, and generally want to run various proprietary games on Windows
> while simultaneously settling for a false sense of security through open
> source firmware wrapping large hidden binary components. Out of sight, out
> of mind, purchase price above all other concerns seems to be the general
> takeaway for that market.

I can't claim to know what the firehose of support tickets is like, but know
that there are users who don't fit this strawman. I'm one of them. I'm not
here to run proprietary games on Windows. My stated goal is to promote the
development of portable software and the adoption of freer systems by using my
board to introduce POWER9 support to builds.sr.ht, giving access to POWER9
cycles to tens of thousands of developers. It's causes like this which help
address the problems you mention of fat software needing to be improved to
make alternative architectures viable. No small part of why Chromium is slow
on ppc and fast on x86 is because there's a JIT for x86 and not for ppc, not
necessarily because of fundamental problems in Chromium's design.

But yeah, there are fundamental problems in our entire ecosystem with respect
to disreagarding performance thanks to the luxuries afforded to us by fast x86
systems. Is the answer to this that we should be writing off alternative
hardware models? No, the hardware should strive to be fast and the software
should strive to meet it. Both parties have a responsibility to performance
and we can't lay the blame at each other's feet and ignore our role in it.
It's a slow process to change the industry, we must exercise patience.

EDIT: it's been pointed out to me that Chromium has a JIT on POWER9. This
doesn't substantially change my response.

~~~
tpearson-raptor
> This reads like blame shifting, not accountability.

I don't think I was clear enough here, sorry. Yes, Raptor Computing Systems is
accountable for its poor support experience. Their management is trying,
unsuccessfully, to manage the support load. I am actually on the engineering
and technology side moreso than the support and business side, and I have no
direct ability to change how support is handled beyond making (strong)
recommendations.

Fundamentally, however, the blame does lie primarily with consumers that are
unwilling to pay for owner controlled computing. Raptor took a risk with the
Blackbird, attempting to broaden the reach of owner controlled compute beyond
business and high end workstation users, and fundamentally underestimated >
20x additional support load that new market would end up causing. Raptor does
not have the resources to handle that kind of support load, and the increases
in product pricing required to do so would put Blackbird back out of reach of
everyone except business users.

Raptor was working on fixing these issues regardless, but AMD Rome has made
that nearly impossible due to the (predictable, but unfortunate) customer
reaction that I detailed earlier. We have had to pivot to a different
mechanism to try to resolve the problems, and I am not allowed to say more
publicly as to what that will be at this time, other than it is coming for
October.

~~~
Sir_Cmpwn
Thank you for clarifying more.

Not sure if you're following IRC, but just in case I'll mention it here as
well: Raptor products are already outpricing most consumers. Even in casual
conversations with hacker friends, most of them balk at the price. Point
being: if you increased your prices most of your buyers wouldn't blink. At
present your buyers _are_ blinking at the poor support experience. I would
have paid a 50% premium for a product with actual support.

A compromise would be to offer ala-carte support packages, perhaps billed
yearly. I would also have paid for that.

I know these decisions aren't in your court, but I don't have anyone else to
share my opinion with, so there you have it. I hope that I will publish a more
positive review of Raptor with these problems addressed soon. Thanks for
coming by the thread to share your thoughts.

~~~
Luke-Jr
A number of users on IRC also said they _would_ be priced out by any 50%
premium increase...

~~~
Sir_Cmpwn
(1) I _would_ have paid 50% more but I also imagine that 50% is hardly
necessary to fund a proper support system. That represents an increase of
nearly a thousand dollars per unit.

(2) Having known the person (singular)* who said they'd be priced out at +50%
from other interactions, I wouldn't take their comments at face value.

* Apparently it's two people.

~~~
sthrs
What you're saying is disingenuous at best and gaslighting at worst. Just
because you apparently have a very generous budget to throw at a problem
doesn't mean everyone else in the same space does. I really struggle to see
what the point is in making such a comment about other people's financial
situations, and I would hope you can either justify making such an
inflammatory remark or walk it back.

~~~
Sir_Cmpwn
The specific person I'm referring to has lost any goodwill from other
interactions I've had with them. I don't trust their statements.

And like I said, I would have personally paid a 50% increase, or for an ala-
carte support package, but I doubt a 50% bump is actually necessary. One which
doesn't push the board out of people's price ranges is probably feasible.
Remember that we're talking about $2-3K, for the cheapest option with the
simplest loadout. Customers already have disposable income. These are not
priced like consumer hardware.

~~~
sthrs
I beg you to reconsider the repercussions of holding grudges, especially in
the FOSS community. All too often I see this pattern where people take a few
disagreements too personally and hold them against each other for eternity. I
know who you're talking about specifically (I was on the IRC channel at the
time) and I know that they are genuinely trying to make it up to you (full
retractions, refraining from speaking ill of the sort in the future,
attempting to adjust their behavior so that they are not so quick to jump to
so-called "hot takes" or other aggressive behavior). I hope that you, in turn,
could also, in your own time, learn to forgive and let go as well. Please, for
the sake of good FOSS everywhere - I know the two of you could accomplish so
much!

------
lostgame
This is unfortunate. I have loved the POWER architecture since Apple’s G5.
(Though I am aware that was a variant.)

------
einpoklum
So, the root of the problem seems to be in the shipping?

I had assumed this would be a story about Power9-based machines...

~~~
Cuuugi
read past the first paragraph.

