I don't have much to say about the other discussion but the article really got my teeth on edge, because of the tabloidy anthropomorphization of something much simpler - "the loss function wasn't regularized properly leading to overfitting!"
Rahul On Mon, Dec 31, 2018 at 9:09 PM Vinit Bhansali <[email protected]> wrote: > It wasn't cheating or doing a poor job. It worked within the defined > parameters. > The parameters themselves were poorly defined. > > Interesting that it always boils down to the this "I have the right answer. > But do you have the right question". > > ---- > This clever AI hid data from its creators to cheat at its appointed task // > TechCrunch > > https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-from-its-creators-to-cheat-at-its-appointed-task/ > > Depending on how paranoid you are, this research from Stanford and Google > will be either terrifying or fascinating. A machine learning agent intended > to transform aerial images into street maps and back was found to be > cheating by hiding information it would need later in “a nearly > imperceptible, high-frequency signal.” Clever girl! > > This occurrence reveals a problem with computers that has existed since > they were invented: they do exactly what you tell them to do. > > The intention of the researchers was, as you might guess, to accelerate and > improve the process of turning satellite imagery into Google’s famously > accurate maps. To that end the team was working with what’s called a > CycleGAN — a neural network that learns to transform images of type X and Y > into one another, as efficiently yet accurately as possible, through a > great deal of experimentation. > > In some early results, the agent was doing well — suspiciously well. What > tipped the team off was that, when the agent reconstructed aerial > photographs from its street maps, there were lots of details that didn’t > seem to be on the latter at all. For instance, skylights on a roof that > were eliminated in the process of creating the street map would magically > reappear when they asked the agent to do the reverse process: > > The original map, left; the street map generated from the original, center; > and the aerial map generated only from the street map. Note the presence of > dots on both aerial maps not represented on the street map. > > Although it is very difficult to peer into the inner workings of a neural > network’s processes, the team could easily audit the data it was > generating. And with a little experimentation, they found that the CycleGAN > had indeed pulled a fast one. > > The intention was for the agent to be able to interpret the features of > either type of map and match them to the correct features of the other. But > what the agent was actually being graded on (among other things) was how > close an aerial map was to the original, and the clarity of the street map. > > So it didn’t learn how to make one from the other. It learned how to subtly > encode the features of one into the noise patterns of the other. The > details of the aerial map are secretly written into the actual visual data > of the street map: thousands of tiny changes in color that the human eye > wouldn’t notice, but that the computer can easily detect. > > In fact, the computer is so good at slipping these details into the street > maps that it had learned to encode any aerial map into any street map! It > doesn’t even have to pay attention to the “real” street map — all the data > needed for reconstructing the aerial photo can be superimposed harmlessly > on a completely different street map, as the researchers confirmed: > > The map at right was encoded into the maps at left with no significant > visual changes. > > The colorful maps in (c) are a visualization of the slight differences the > computer systematically introduced. You can see that they form the general > shape of the aerial map, but you’d never notice it unless it was carefully > highlighted and exaggerated like this. > > This practice of encoding data into images isn’t new; it’s an established > science called steganography, and it’s used all the time to, say, watermark > images or add metadata like camera settings. But a computer creating its > own steganographic method to evade having to actually learn to perform the > task at hand is rather new. (Well, the research came out last year, so it > isn’t new new, but it’s pretty novel.) > > One could easily take this as a step in the “the machines are getting > smarter” narrative, but the truth is it’s almost the opposite. The machine, > not smart enough to do the actual difficult job of converting these > sophisticated image types to each other, found a way to cheat that humans > are bad at detecting. This could be avoided with more stringent evaluation > of the agent’s results, and no doubt the researchers went on to do that. > > As always, computers do exactly what they are asked, so you have to be very > specific in what you ask them. In this case the computer’s solution was an > interesting one that shed light on a possible weakness of this type of > neural network — that the computer, if not explicitly prevented from doing > so, will essentially find a way to transmit details to itself in the > interest of solving a given problem quickly and easily. > > This is really just a lesson in the oldest adage in computing: PEBKAC. > “Problem exists between keyboard and computer.” Or as HAL put it: “It can > only be attributable to human error.” > > The paper, “CycleGAN, a Master of Steganography,” was presented at the > Neural Information Processing Systems conference in 2017. Thanks to Fiora > Esoterica and Reddit for bringing this old but interesting paper to my > attention. > > > - Vinit > ---- > > Read in my feedly.com >
