It looks like we are actually very far. I don’t think, e.g., that there is a single implementation of an end-to-end dialog system that learned to do useful things. For example, how do you train a system that will book hotels, flights, rental cars, shop on Amazon etc… It needs to read forms, fill them and do a lot of things. If humans can decompose these tasks into much smaller one that heavily rely on classification, collect and clean data, architecture training pipelines, only then deep learning comes into play.
It’s interesting, but exactly the same thing could be said of humans:
– they are not easy to implement and maintain: their aggregated data cannot be easily interpreted by the average engineer and algorithms (if any) are not easy to implement and test;
– they are not updateable on the fly;
– they are not efficient at query time: queries should be fast.
By contrast, humans *do* satisfy the two criteria you dropped from the original version of this post:
– they work with little data if needed;
– they are accurate within reason.
The hype is heavy indeed. My framework for thinking about this is closed-world problems vs open-world problems.
Machine learning is closed world, artificial intelligence, is, by definition, open-world: dictionary defines intelligence as “ability to react in new situations”.
Closed-world problems (like chess or go) can and will be beaten into submission through massive power learning very wiggly models – the problem space will be sufficiently densely explored so that it becomes an interpolative ML problem dealing with “old situations”.
Open-world problems (like driving a car, or designing a meaningful data warehouse schema) will not be solved in the foreseeable future. They are extrapolative, *-intelligence problems with a large “new situation” content. The best we have on “general AI” is Douglas Hofstadter stuff, really.
The open-world character in conjunction with the number of 9s required is one of the reasons self-driving cars will not be in-production for another at least 10 years.
I’m happy to see that you didn’t mention IBM in the list of companies. Extraordinary disclaimer: I work for a company that would like to be like IBM. I have a PhD in AI, but not in machine learning, so maybe it’s not AI by current standards/hype.
Laptops are made for laps. Awkwardness is probably the most salient characteristic of academics.
My not-as-educated-as-I’d-like-to guess is that DL works, but it’s not the end of the road for AI. I hope the next steps don’t get patented…
If you want to optimize performance in data engineering, put a bit of old AI (logic) and you will have a lot of optimization to do, to make it work again.
WRT AGI, IMHO it’s the 90% syndrome. The data used is becoming bigger and bigger, we may not be getting closer at all. On the negative side, AI would be the last problem that humanity needs to solve. On the positive one, we still have to figure out many things before that day, like a way for economic transition.
Michel Lemaysays:
Nice mind-wandering post! It was nice to meet you [again] in a summit somewhere.!
One thing that always amaze me with deep learning is the fact that, as you mention it as well, is so brute force. What an awful amount of joules and compute dollars wasted over and over just replicating existing experiments. Can we share theses hidden layers? Can we share theses building blocks? Currently we are directly aimed at an oligopoly where only big corporations or select few research centers can afford to train theses architectures.
The end goal of that always seems to be human-level intelligence but are we gonna be able to reach it in a world of limited resources?
Michel: When I started my post, I was planning to include our meeting… it would have made the story even nicer… maybe next time! We ought to coordinate to make sure to meet again!
Regarding power usage… We cannot build a drone that can fly as well as a bee. Yet we know that it is possible to do so while using very little power (a bee’s brain uses barely any power at all) and very little training. This suggests that scaling up will require more than just brute force.
Dmitry Ganyushinsays:
“The standard advice I provide is not to trust the academic work.” – very nice!
Chrissays:
I strongly recommend Hinton’s talk on what is wrong with convolutional neural nets:
The essential complaint is “yes, it kind of works, if you have enough data, but we need an approach that is not so data hungry”. He has a long, detailed and compelling example about vision, and shows what it would take to build something that has better long-term prospects, at least for vision. Part of the argument is that people do not need to see an object from all kinds of different angles before learning to recognize it, but current machine learning algorithms do need this. Something is being missed, and Hinton has a suggestion for what this is.
The reason I like this talk so much is that Hinton really knows the biology as well as the engineering, thinks deeply about the task, and has no need to indulge in hype.
It looks like we are actually very far. I don’t think, e.g., that there is a single implementation of an end-to-end dialog system that learned to do useful things. For example, how do you train a system that will book hotels, flights, rental cars, shop on Amazon etc… It needs to read forms, fill them and do a lot of things. If humans can decompose these tasks into much smaller one that heavily rely on classification, collect and clean data, architecture training pipelines, only then deep learning comes into play.
I agree though I would not dare to put a time frame. It could be that we are one week away from the breakthrough we lack.
I agree, predictions are hard.
Lovely post.
It’s interesting, but exactly the same thing could be said of humans:
– they are not easy to implement and maintain: their aggregated data cannot be easily interpreted by the average engineer and algorithms (if any) are not easy to implement and test;
– they are not updateable on the fly;
– they are not efficient at query time: queries should be fast.
By contrast, humans *do* satisfy the two criteria you dropped from the original version of this post:
– they work with little data if needed;
– they are accurate within reason.
You are a bit too clever, especially because you went to see my original paper and found out that I quoted myself selectively.
Just taking my commentary to the next level. 🙂
The hype is heavy indeed. My framework for thinking about this is closed-world problems vs open-world problems.
Machine learning is closed world, artificial intelligence, is, by definition, open-world: dictionary defines intelligence as “ability to react in new situations”.
Closed-world problems (like chess or go) can and will be beaten into submission through massive power learning very wiggly models – the problem space will be sufficiently densely explored so that it becomes an interpolative ML problem dealing with “old situations”.
Open-world problems (like driving a car, or designing a meaningful data warehouse schema) will not be solved in the foreseeable future. They are extrapolative, *-intelligence problems with a large “new situation” content. The best we have on “general AI” is Douglas Hofstadter stuff, really.
The open-world character in conjunction with the number of 9s required is one of the reasons self-driving cars will not be in-production for another at least 10 years.
I’m happy to see that you didn’t mention IBM in the list of companies. Extraordinary disclaimer: I work for a company that would like to be like IBM. I have a PhD in AI, but not in machine learning, so maybe it’s not AI by current standards/hype.
Laptops are made for laps. Awkwardness is probably the most salient characteristic of academics.
My not-as-educated-as-I’d-like-to guess is that DL works, but it’s not the end of the road for AI. I hope the next steps don’t get patented…
Relevant: https://xkcd.com/1838/
If you want to optimize performance in data engineering, put a bit of old AI (logic) and you will have a lot of optimization to do, to make it work again.
WRT AGI, IMHO it’s the 90% syndrome. The data used is becoming bigger and bigger, we may not be getting closer at all. On the negative side, AI would be the last problem that humanity needs to solve. On the positive one, we still have to figure out many things before that day, like a way for economic transition.
Nice mind-wandering post! It was nice to meet you [again] in a summit somewhere.!
One thing that always amaze me with deep learning is the fact that, as you mention it as well, is so brute force. What an awful amount of joules and compute dollars wasted over and over just replicating existing experiments. Can we share theses hidden layers? Can we share theses building blocks? Currently we are directly aimed at an oligopoly where only big corporations or select few research centers can afford to train theses architectures.
The end goal of that always seems to be human-level intelligence but are we gonna be able to reach it in a world of limited resources?
Michel: When I started my post, I was planning to include our meeting… it would have made the story even nicer… maybe next time! We ought to coordinate to make sure to meet again!
Regarding power usage… We cannot build a drone that can fly as well as a bee. Yet we know that it is possible to do so while using very little power (a bee’s brain uses barely any power at all) and very little training. This suggests that scaling up will require more than just brute force.
“The standard advice I provide is not to trust the academic work.” – very nice!
I strongly recommend Hinton’s talk on what is wrong with convolutional neural nets:
(https://www.youtube.com/watch?v=rTawFwUvnLE)
The essential complaint is “yes, it kind of works, if you have enough data, but we need an approach that is not so data hungry”. He has a long, detailed and compelling example about vision, and shows what it would take to build something that has better long-term prospects, at least for vision. Part of the argument is that people do not need to see an object from all kinds of different angles before learning to recognize it, but current machine learning algorithms do need this. Something is being missed, and Hinton has a suggestion for what this is.
The reason I like this talk so much is that Hinton really knows the biology as well as the engineering, thinks deeply about the task, and has no need to indulge in hype.