This tracks with my feelings making and using Stable Diffusion Loras and fine tunes. Still, with the speed to train and use, Loras have worked for me in most use cases and it hasn't been worth fine tuning the entire model.
Yeah,it reflects the “feel” I get from lLoRa as well, especially if I overdo it. The new data becomes the preferred output even for unrelated inputs. I always felt like it was bludgeoning the model to some extent vs finetuning.
Also, LoRa tuning an extensively tuned model occasionally provokes full on delusional “insanity” or gibberish seizures.
I have had really good luck though using a highly tuned model as the training basis for a LoRa and then applying that LoRa mask to the base version of that model. I’m not sure why that seems to work better than the same LoRa training directly on the base model.
I've done a lot of tinkering with the internals of LoRA training, specifically investigating why fine-tune and LoRA training result in such different results, and I'm no academic, but I have found that there are definitely some issues with the SOTA at least WRT Stable Diffusion.
I've had significant success with alternate init mechanisms (the standard technique of init'ing B to zeros really does hurt gradient flow), training alpha as a separate parameter (and especially if you bootstrap the process with alphas learned from a previous run), and altering the per-layer learning rates (because (lr * B) @ (lr @ A) produces an update of a fundamentally different magnitude than the fine-tune update of W * lr = lr * B @ A).
In the context of Stable Diffusion specifically, as well, there's some really pathological stuff that happens when training text encoders alongside the unet; for SD-1.5, the norm of "good" embeddings settles right around 28.0, but the model learns that it can reduce loss by pushing the embeddings away from that value. However, this comes at the cost of de-generalizing your outputs! Adding a second loss term which penalizes the network for drifting away from the L1 norm of the untrained embeddings for a given text substantially reduces the "insanity" tendencies. There's a more complete writeup at https://github.com/kohya-ss/sd-scripts/discussions/294#discu...
You also have the fact that the current SOTA training tools just straight up don't train some layers that fine-tunes do.
I do think there's a huge amount of ground to be gained in diffusion LoRA training, but most of the existing techniques work well enough that people settle for "good enough".
Most people are using LoRAs as a solution for IP transfer.
Thing is Ideogram v2 has already achieved IP transfer without fine tuning or adapters. So we know those aren't needed.
Is Ideogram v2 an exotic architecture? No, I don't think so.
Are there exotic architectures that will solve IP transfer and other tasks? The Chameleon and OmniGen architectures. Lots of expertise went into SD3 and Flux dataset prep, but: the multimodal architectures are so much more flexible and expressive.
Flow matching models are maybe the last we will see before multi-modal goes big.
What to make of things in the community? How is it possible that random hyperparameters and 30 minute long fine tunings produce good results?
(1) Dreambooth effect: if it's like, a dog, you won't notice the flaws.
(2) Filing drawer problem. Nobody publishes the 99 things that didn't work.
(3) SD <3 struggled with IP transfer on image content that could not have possibly been in its datasets. But laypeople are not doing that. They don't have access to art content that Stability and BFL also don't have access to.
(4) Faces: of course SD family saw celebrity images. Faces are over-represented in its datasets. So yeah, it's going to be good at IP transfer of photographic faces. Most are in-sample.
I've been rolling the idea around that perhaps if a product is encumbered by a subscription then it's not a first sale and the product counts as inventory. And gets taxed as such.
I don't know the first thing about such things, but I'll bet if companies were pushed on this they'd suddenly start asserting that it's a lease, which I think is typically taxed similarly to a sale.
I hate the "everything's a subscription" business model that's taking over everything. We'll achieve peak serfdom when the air we breathe, water we drink, and food we eat is bought on a subscription model.
Food sometimes is, and honestly a more centralised food subscription system could drive down the cost of food by making demand more predictable and enabling better economy of scale.
If it were feasible to charge you for your own heartbeats, companies absolutely would. Infinite growth is a poison pill idea - it brought us to where we are today, and would drag us to the future you outlined, unless the folly is abandoned.
Nothing wrong with growth,it's natural. The population grows and so does the money supply in turn. This is good. What is not good is when a particular company is trying to retain ALL growth of certain segments for themselves through abusive leverage.
A principle that stuck out to me as a child was that our society prioritized lower prices for consumers as a whole over the prosperity of any one company or industry.
We let companies grow to the point where they now subvert the will of the people.
Growth is good until it's not, cancer would like a word.
The only thing that seems capable of growing forever is the amount of space in the universe. Line go up increasingly fast always forever is a bullshit fever dream that needs to be culled with prejudice.
Not in real estate. There is technically no difference between leasing and renting, other than the things that are colloquially associated with. A lease agreement (or rental agreement) may or may not have a clause offering the borrower an opportunity to take ownership.
Rent might also refer to the price portion of a leasing agreement (or rental agreement). So in shorthand usage, “the lease” might encompass all terms of borrowing something, including the price, but “the rent” might just refer to the price.
Does it work that way for ISPs that "lease" their modems? This is the first time I've ever considered this idea that the hardware would still be inventory. Does inventory get taxed annually or monthly? Seems like the monthly lease fees would more than cover that.
That's a perfect example. They demand it back at the end of your contract, there's not even an option to keep it, let alone modify it.
I remember reading that, back in the 50s or 60s, the phone company owned "your" phone. It was permanently attached to the wall, and you weren't allowed to do anything to alter it. Did AT&T pay taxes on those phones as inventory?
That's a bit hyperbolic. It was just hung on the wall. If it was permanent, they wouldn't be able to take it back. Thanks for the reminder that we're to the point in time that "kids today" honestly have no memory of land line phones.
But yes, the phones were only available from Ma Bell, and you did have a monthly fee for them. They did have table top versions as well, so it wasn't just wall mounts. They were heavy solid well built devices. Once it was opened for anyone to make, they became cheap light weight plastic pieces of crap.
If one was not returned, they charged what it cost to replace it.
Wish I knew about inventory. My guess is yes! They would have to do accounting on all the modems anyway. Service, where they are all at, serial numbers and more.
When one went bad, it usually involved either a visit to the ISP, or a tech shows up with a new one and a few records get updated and the modem ends up home.
Cable TV boxes are another example!
I do know those were inventoried. A friend went to work as a tech for a local cable company. (Yes, they did tell me how to enable decryption on all channels for at least one model...) They fixed the incoming boxes and those units went right back out to homes.
The units were purchased a few times per year to balance subscriber growth and attrition. Good repair metrics saved a TON of money. The units were from one to a few hundred dollars a pop!
Rental was $7.99 / month at one point I could remember.
Say the box was $150, at that rental rate the box becomes a minor profit item after a year and a half, right?
Well, most cable boxes got used a decade or more, especially when the company did not have to change its encryption tech.
That is a 4 to 5x return on the purchase price. Maybe 3x return after attrition, failed returns and repair costs get woven in.
Not a bad deal for them. And as the user, not many worries, but also basically zero options.
An example from cable boxes was serial ports and input and output ports either disabled or nerfed to the point of poor usability really sucked for anyone wanting to use the gear technically, or as part of an automation of some kind.
They did! And the not so nice part of that was being stuck with the phones they supply. Now, as simple phones, I liked them. Audio quality was great, the device did not break easily and all that. No worries. But, if you wanted to do anything connected to that phone, or the line, that was off limits!
The nice part was it being their phone, they took care of it. Service was a call away and the service tech could do what it takes and have few worries.
Begin sidebar:
Our wall mounted phone was surrounded by phone numbers and other bits of information. I started it one day writing the name of a burger joint we used to call all the time by the phone.
Mom was pissed, but Dad liked it! Next time we went to call the number was right there!
And so it began...
When I left home for the last time, I looked at that phone and wall for a long time. Many years of our lives were there. Friends, family, businesses, other things like EMS, poison control, various hotlines were all there organized fairly well given the organic way it started.
I wish I had taken a picture!
Seeing that happen and being a part of it all is probably one of the more potent reminders, to me of course, of what the pre-digital times were like.
- If a consumer buys a car with heated seats and an option to activate the heated seats as a subscription, but the consumer elects not to subscribe, do the inactive heaters still count as inventory? If so, what if the car or heaters get destroyed but the automaker doesn't ever learn about the destruction?
- Would this apply to e-books and media? In today's market, if I buy an e-book or media from a streaming service, I'm not buying a copy, but rather a revocable license with a one-time fee. It seems like that e-book or media is inventory for the book seller.
IMHO it should be on the automaker to learn about the destruction. They should have to carry it as inventory just like anything else. There should be real expenses to it, not just the ability to throw it out there with some tech that blocks it unless a user registers and pays the subscription, and not have to do anything about it. I think they should also be required to pay for repairs if/when it breaks. If they want to retain ownership of the heated seats so they can charge a subscription for it, then they need to own (pun intended) the responsibilities of ownership.
While I do think that's fair, my personal hope is that such a requirement would make them stop this asinine practice.
When I buy something I must pay sales tax to the government. Later, if it turns out I never owned it and ownership is revoked, can I get the sales tax back?
Yeah, it's not height itself that's the problem, it's body shape. As just one example: some people have long torsos and short legs, others long legs and short torsos. Those two groups will have wildly different BMI curves even at similar health levels.
BMI is a very crude measure. It is only useful in the extreme, and not particularly relevant for an individual. If you have a BMI of 40 you should think about losing weight, but exactly nobody need a composite number to know they are that fat. Similarly, you might have a normal BMI but an LDL of 300 or a strange lump on your thyroid. These things are vastly more important than your BMI
While I love the use of passive aggressive language, in this case, as it was originally written, "However, if you are not interested in factual information, the answer is: YES" causes some globule of confusion to the primary reader --
Is this to imply that the answer Yes to the question about Matt's control over the organizations named WordPress is _not_ in fact factual?
At my theater some people used to get nude too. RIP Rialto Theatre - to add insult to injury it’s a church now. Dr. Frank-N-Furter is rolling in his grave.
So, the first time I went to see it I was there by chance, because it was in an amusement park and I really didn't know the first thing about what I was getting into.
And yes, at the time I thought the people were being rude, especially when they where howling at the usherette.
Then I saw other performances online and felt like a complete tool :)
