AI can't invent. AI isn't creative.

[ivconic]

The worst are "languages" that offer instant portability (Python is a good example), but even worse are web-based "languages" (it's not really a real language).
I think the future will remain only on hardware-oriented software.
Everything else will crumble like a house of cards.

[moodz]

I agree ... Programming languages are built for humans not machines. AI would rather write 1000 lines of C rather than a single abstraction statement. Optimisation can't be done efficiently when you are far from the machine logic. I noticed recently that AI is even better at HDL than it is at procedural coding because HDL like verilog or VHDL is very close to the actual chip logic. Historically this has made good HDL difficult for humans but AI eats it for breakfast.

[Aziz]

Yep, I agree too.

That's the reason, why I rather fall back into pure ANSI-C coding (procedual programming). Even avoiding C++.
The pure reason for using C++ is the memory management of objects and structures (allocation, init, free, life cycle, exception handling). Not managing object orientation (OO). All the OO features will lead sooner or later to quirk dev.

BTW, the core implementation of PHP is written in C.

One important thing to know:
The compiler optimization (re-ordering the machine code) can add some serious vulnerability to the execution code too. Even there is no issue with the source code.

[Aziz]

BTW,

when a software product grows over the years, it also ends mandatory in quirk dev.
Copy & paste and quick & dirty dev also leads to quirk dev.
I know this, because I have a master grade in quirk dev.

I have ported many old quirk 16-bit MS-DOS apps into Windows apps.
And I have found many quirks too.

As the computer stuff technology is steadily changing (developing), only change endures.
So it must end sooner or later in quirk dev.

[Aziz]

Back to AI:

The extreme quantization of the AI parameters (weights) down to three states/values (+1, 0.0, -1):
n = ld(3) = 1.58 bits.

https://en.wikipedia.org/wiki/1.58-bit_large_language_model

This is the minimum available quantization level at the moment.
Proof:
An information quantity may be strengthened (+1), destroyed (0.0), or inhibited/negated (-1). This is the base of all atomic information processing operation.

Storing 1.58 bits in a computer memory is difficult. So it ends into 2 bit. We will be available to run all the trillion parameter AI models straight out of the local memory of the PC soon.

I am sure, we will end up in quantization level of 1 bit sooner or later (1=strength or 0=destroy information quantity).

[Aziz]

Found an interesting article on 1-bit-quantization:

https://www.mindstudio.ai/blog/1-bit-quantization-cactus-bonsai-27b-model-phone

Reducing an information quantity into +1 (strength), -1 (negate) states does not solve the problem yet.
Destroying an information quantity (with 0) is one of the most important operations in AI models.

Any irrelevant input information must be destroyed (independent to the problem solution) .
If you feed a noise input to the neural network and train it to a task, you will see, that all the weights referring to the noise input will be trained to 0 at the end.
So the neural network will destroy the input information at the beginning.

Strength (+1), destroy (0) and negate (-1) an information quantity stays very important. Yet.

[Aziz]

Storing 3-state information into single cell memory will lead to special NPU chips.

https://en.wikipedia.org/wiki/Multi-level_cell

There is no other way until we find a good solution for 2-state (1-bit) information processing.

This is a good opportunity for winning the Nobel Prize.
Go and get it. I believe in you.

[ivconic]

Let's take a quick break for a laugh!
Whenever we talk about AI; I can't help but remember this:

[eduardo1979]

[ivconic]

I've been using an iPhone for years, various versions.
When I'm in the mood for a fight; I mostly spend my time arguing with Siri!
Ha, ha, ha!
Sometimes these arguments reach the level of heavy swearing!
A good "anti-stress" program.
Exhaust valve!
And then as if born again; I am able to behave civilly with the people around me!

[Aziz]

, nice!

BTW,

the training of LLM's is making huge progress. In the future, we will get much faster and better model updates.
Dust (brand new training method)

An AI paper shows the comparison of traditional backpropagation vs. dust training.

[Aziz]

Hi all,

Dust is approx. 3 weeks old. This is a real break-through AI technology.

In all the AI labs, there is much "dust" now.
Caused by stress, hurry, "gold fewer", implementation and optimization pressure .. to get the best AI model ever.


We give them a few month time. So the "dust" can settle down.

Aziz

PS: added links to Dust

https://qlabs.sh/research/dust

https://github.com/qlabs-eng/dust

[moodz]

Mr Claude ran a full comparison of Dust vs Backprop on my local machine .... I am afraid that Dust must be renamed to Smoke ( and Mirrors ).

The method does work BUT ... Dust uses alot more memory and the training takes longer.

[Aziz]

Hi Paul,

thanks for the nice comparison made by Mr. Claude.
Mr. Claude has definitely more IQ than Mr. Gemini (Google).

Oh well, the current chips aren't designed for dust. They are highly optimized for backprop.
At the moment, it is really smoke & mirrors.
We have to wait and see what happens.

My first Backprop algorithm implementation dates to 1992 (or so) during my studies.
Scientists didn't find a better solution for machine learing up to date obviously.

"Claude, invent me a better machine learning algorithm which beats Backprob.."
Aziz

[Aziz]

Hi all,

the AI is doing a great job in the hands of Paul right now.
We will see some interesting news I think soon.
But it is up to Paul to do this.

Aziz