Where do you get the real data, though? They just scrap data from websites
Great question… Do they “just” scrape data from websites?
https://www.theatlantic.com/technology/archive/2025/03/libgen-meta-openai/682093/
Keeping it clean would require hiring people to scrub contamination from the data sets.
That’s exactly right.
The point I was trying to raise that wasn’t semantics was that if the majority of the full training data were synthetic, it could lead to model collapse.
But luckily (or not?) a small amount of finetuning can be very effective in correcting the range of responses.
Open LLMs are finetuned on partially or fully synthetic data all the time
That’s what I was suggesting.
You explained to me you weren’t talking about “finetuning”, but training on completely synthetic data.
(Fine-tuning happens after the LLM has already been trained)
Oh, yeah then I agree with above commenter. This would collapse the model.
You’re not training an LLM on text generated by an LLM. You’re training it on 98% real data, and intentionally biasing it by sprinkling in the fake data intermittently.
But times change and the cost of free tier users surpasses that of paying users. Should the company continue providing the same level of service for free tier users?
“Times changing” here seems to be the central trick to the argument.
What’s interesting about enshittification is that as the company gets more and more profitable there seems to be more and more excuses as to why these free features are so costly.
It’s very easy for a company to put out a statement that times are changing and that the free tier is unaffordable. Is that always true? Who’s to say?
I’m sure sometimes it is true but the doubt is why arguments like this will never go away.
Also, what other term than entitlement would you use for somebody gets something for free, is not promised that it will stay free forever, the free offering is cancelled or limited, and the user starts complaining?
What other term than incompetent would you use for a company that puts out a free product, attracts a bunch of free users, abruptly cuts access for those features and puts it behind a paywall, and then acts surprised when those same users complain about it.
If you want to make a business move go ahead, it’s your right, but accept the complaints from your user base you predictably pissed off.
People aren’t interested in “learning about LLMs”, especially people like artists.
They’re interested in telling Elon Musk to “fuck off”, and when Grok says something bad about Elon it’s very cathartic for them.
They might know it’s feeding their own thoughts back to them, but they don’t care. To people who aren’t in the know, this box Elon is promoting as “objective truth box” is criticizing Elon. That’s a very powerful narrative in a world where he’s taking over the world.
It’s hard to disagree. Elon can go fuck himself. What’s more important to the average person, stopping Elon or understanding the nitty gritty of machine learning?
When artists say AI is stealing, they’re not interested in an explanation about how “its really not”. And if you tried to, they’d feel you’re missing the forest for the trees because their problem with AI isn’t metaphysical philosophy, it’s that it’s hurting their job opportunities.