What’s Really Going On in Machine Learning? Some Minimal Models | Stephen Wolfram | August 22, 2024
What’s Really Going On in Machine Learning? Some Minimal Models | Stephen Wolfram | August 22, 2024
Stephen Wolfram explores minimal models and their visualizations, aiming to explain the underneath functionality of neural nets and ultimately machine learning.
This was a terrible article from a serial plagiarist who refuses to do work or cite sources.
But at a fundamental level we still don’t really know why neural nets “work”—and we don’t have any kind of “scientific big picture” of what’s going on inside them.
Neural networks are Turing-complete just like any other spreadsheet-style formalism which evolves in time with loops. We've had several theories; the best framework is still PAC learning, which generalizes beyond neural networks.
And in a sense, therefore, the possibility of machine learning is ultimately yet another consequence of the phenomenon of computational irreducibility.
This is masturbatory; he just wants credit for Valiant's work and is willing to use his bullshit claims about computation as a springboard.
Instead, the story will be much closer to the fundamentally computational “new kind of science” that I’ve explored for so long, and that has brought us our Physics Project and the ruliad.
The NKoS programme is dead in the water because — as has been known since the late 1960s — no discrete cellular automaton can possibly model quantum mechanics. Multiple experts in the field, including Aaronson in quantum computing and Shalizi in machine learning, have pointed out the utter futility of this line of research.
Permanently Deleted
I don't know a lot about AI or machine learning so I'd take what I say with a grain of salt. I do know a lot about computers, though. I'm just spit balling here.
This is kinda the reason why I think this "AI" hype is a joke. I get the idea behind it, but a computer is only as smart as the user. Or in this case the data that it soaks up. And as advanced as they are they are mostly still just a novelty save for very specific purposes. The whole idea of a black box in machine learning is just inefficient and wasteful. The fact that we have no idea how these AI's achieve their output is a big problem and a huge waste of resources. In a basic sense, if you put 2+2 into a calculator it will give an output of 2. If you put 2+2-(3x9-18)+7 into a calculator it will give you an output of 2. If all you see is the result you will have no idea how much processing power is being wasted on unnecessary processes. As long as we keep shoving information into these things without thinking about what we put into them they will only get more wasteful with unnecessary data. I know they add certain parameters and weights to negate things like this. But there's no way in hell they've accounted for even 1% of what would be needed.
Don't get me wrong, I understand the practicality of using machine learning. I just think the way we are building it from the ground up is too simple for what we are trying to achieve at this point. I honestly think we are reaching a plateau with this kind of machine learning. We need more parameterization if we want it to get better.
I don’t know a lot about AI or machine learning so I’d take what I say with a grain of salt.
Yeah...
More data doesn't mean more processing.
They have a fixed number of interconnected nodes that encode the data via the weights between them.
So processing requirements are the same. Training is where a lot of power goes though.
This also gives it the ability to solve things we don't have an equation for. As long as we know the input and output we can train an NN to do the calculations even if we don't know the equation ourselves.
The black box nature is a problem but that's also where it's power comes from.
I see what you mean. I made a comment further down as a response to someone else where I go into more detail about my train of thought where I explain more of the issues I've found with this type of machine learning.
What do you think about what Steven Wolfram wrote here?
I've only made it about half way through as of right now. Like I stated previously, I'm not an expert. But I still believe in my statement above. From what I can gather it takes extreme amounts of effort to even figure out how a neural network arrived at the conclusion it came up with. And that still seems like a backwards way of approaching it. You're starting at the end and working your way backwards. It's not a bad method. Just in my opinion I believe it's the wrong method.
If you compare this to something like a scientific theory it doesn't quite match up to the procedure. Things like gravity exist regardless of the formulas used to determine it. Because of this we were able to figure out the formulas required to calculate it. With neural network's we already know how they do what they do because we are the ones that programmed them, at least initially. I find it more analogous to solving for the velocity of an object falling over and over again. You already know the formula, so it's relatively easy. You just work your way backwards from the result. Sure we can add things like drag, friction, and terminal velocity to add more parameterization and make it more accurate. But even with something like this, the increase in accuracy slowly decreases while the processing power increases.
Basically what I'm trying to say is that I believe the "formula" is not quite correct. It's adjacent to what we are looking for. You can keep making the initial conditions as complicated as you want but eventually you will reach the realistic computational limitations of said conditions. If the initial conditions are not quite correct, you can only get it within a certain degree of accuracy before it starts to either diverge or plateau.
I'm not saying neural network's are wrong. I'm saying that we are making the wrong kinds of neural network's. Instead of forcing massive amounts of data into these things until we get the result we want, we should try and find more ideal initial conditions that are more equipped to solve the problems we are trying to solve. As I mentioned above the result doesn't matter when the method of solving has unnecessary, or even incorrect, steps involved in the processes.
I am a layman when it comes to neural networks and machine learning, as I stated above. But this is what I see whenever I hear about this kinda stuff. It just all seems so wasteful because we are so focused on the results. It feels like a confirmation bias when we see the results we were expecting so we ignore the underlying issues. If the "black box" is causing issues it seems entirely more likely that it was set up for failure. If you were calculating a theoretical pendulum and it starts doing 360 no scopes instead of going back and forth then the laws and conditions assigned to it were incorrect.
Edit: Added some stuff in case I explained my train if thought poorly.
I think that the mistake is thinking that "smart" is a meaningful word. I'd encourage you to learn about the technology you're critiquing and not listen to memetic bullshit from articles like the one we're discussing. Consider:
- AI/cybernetics/robotics (same field, different perspectives) is always only useful for specific tasks, never for general replacement of humans
- Black-box treatments of machine learning are only done at the most introductory level and there are several ways to examine how e.g. a Transformers-based language model's weights contribute to its outputs
- We have many useful theories about how to learn functions in general, with machine learning as a special case
This has happened before and it will happen again. I'm sure you've seen the phrase "AI winter" floating around.
I encourage you to read the comment I left down lower in response to someone else. I go into more details about the point I'm trying to make. As I mentioned above I'm not an expert on the topic. But I am open to criticism. It's kinda the reason I make comments like this in the first place. I'm not trying to rile people up but get a discussion going.
I appreciate your feedback. I'm learning as I go