Artificial Intelligence - A Guide for Thinking Humans Melanie Mitchell

 

Part One Background

Chapter 1 The Roots of Artificial Intelligence

• The concept of supervised learning is a key part of modern AI.
• Training set. Test set.
• Perceptron learning algorithm.
• Even at beginning, AI suffered from a hype problem approach.
• "Easy things are hard" dictum: the human workers are hired to perform the "easy" tasks that are currently too hard for computers.

Chapter 2 Neural Networks and the Ascent of Machine Learning

• rebutted: réfuter
• to soar: monter en flèche
• pep: vitalité
• imbued: imprégné
• disparagingly: de façon désobligeante

• "activation"
• "back-propagation"
• I trained both a perceptron and a two-layer neural network, each with 324 inputs and 10 outputs, on the handwritten-digit-recognition task, using sixty thousand examples, and then tested how well each was able to recognize ten thousand new examples.
• The term "connectionist" refers to the idea that knowledge in these networks resides in weighted connections between units.

Chapter 3 AI Spring

• overlord: chef suprême
• eerily: sinistrement
• pun: jeux de mots
• derogatory: désobligeant
• tout: racoler
• shallow: superficiel
• self-awareness: conscience de soi
• dogged: obstiné
• contrivance: dispositif
• ergo: par conséquent
• to imbue: imprégner
• to spur: inciter
• to scoff at: se moquer de
• glaringly: extrêmement
• malevolent: malveillant
• to ascribe: attribuer
• wryly: ironiquement 
• rapture: extase
• post-haste: en toute hâte 
• conscripted: appeler
• dire: pressant
• to relent: céder
• a scant: peu de
• surfeit: excès
• to straddle: enfourcher
• zaniest: loufoque
• ploy: stratagème
• stung: piqûre
• wagering: pari
• foil: faire-valoir
• wherein: où
• harbinger: messager
• bearing: position

• The terms narrow and weak are used to contrast with strong, human-level or full-blown AI (sometimes called AGI, or Artificial General Intelligence).
• We're back to the philosophical question I was discussing with my mother: is there a difference between 'simulating a mind' and 'literally having a mind'?
• Ray Kurzweil, who is now director of engineering at Google.
• Singularity: "a future period during which the pace of technological change will be so rapid, its impact so deep, that human life will be irreversibly transformed".
• Kurzweil agrees: "most of the brain's complexity comes from its own interaction with a complex world. Thus, it will be necessary to provide an artificial intelligence with an education just as we do with a natural intelligence.
• Kurzweil's thinking has been particularly influential in the tech industry, where people often believe in exponential technological progress as the means to solve all of society's problems.
• Kapor: "Perception of and physical interactions with the environment is the equal partner of cognition in shaping experience...Emotions bound and shape the envelope of what is thinkable"
• Crucial abilities underlying our distinctive human intelligence, such as perception, language, decision-making, common sense reasoning and learning.

Part Two Looking and Seeing

Chapter 4 Who, What, When, Where, Why

• The neural networks dominating deep learning are directly modeled after discoveries in neuroscience.
• Today's most influential and widely used approach: convolutional neural networks, or (as most people in the field call them) ConvNets.
• This calculation - multiplying each value in a receptive field by its corresponding weight and summing the results - is called a convolution. Hence the name "convolutional neural network".
• Would you like to experiment with a well-trained ConvNet? Simply take a photo of an object, and upload it to Google's "search by image" engine. Google will run a ConvNet on your image and, based on the resulting confidences (over thousands of possible object categories), will tell you its "best guess" for the image.

Chapter 5 ConvNets and ImageNet

• bunch: groupe
• updended: renversé
• jolt: soubresaut
• snooping: fouiner
• terse: sec
• stuffed: empaillé
• to tease out: extraire

• Yann LeCun, the inventor of Convnets.
• A cardinal rule in machine learning is "Don't train on the test data". It seems obvious.
• It turns out that the recent success of deep learning is due less to new breakthroughs in AI than the availability of huge amounts of data (thank you, internet!) and very fast parallel computer hardware.
• Facebook labelled your uploaded photos with names of your friends and registered a patent of classifying the emotions behind facial expressions in uploaded photos.
• ConvNets can be applied to video and used in self-driving cars to track pedestrians, or to read lips and classify body languages. Convnets can even diagnose breast and skin cancer from medical images, determine the stage of diabetic retinopathy, and assist physicians in treatment planning for prostate cancer.
• It could be that the knowledge needed for humanlike visual intelligence - for example, making sense of the "soldier and dog" photo at the beginning of the previous chapter - can't be learned from millions of pictures downloaded from the web, but has to be experienced in some way in the real world.

Chapter 6 A Closer Look at Machines That Learn

• to veer: virer
• speckling: moucheté
• repellent: répulsif
• skewed: faussé
• inconspicuous: qui passe inaperçu
• whack-a-mole: jeu du chat et de la souris
• adversarial: conflictuel
• ostrich: autruche

• I'll explore how differences between learning in ConvNets and in humans affect the robustness and trustworthiness of what is learned.
• The learning process of ConvNets is not very humanlike.
• The most successful ConvNets learned via a supervised-learning procedure: they gradually change their weights as they process the examples in the training set again and again, over many epochs (that is, many passes through the training set), learning to classify each input as one of a fixed set of possible output categories.
• Demis Hassabis, co-founder of Google DeepMind.
• Deep learning requires big data.
• Have you ever put a photo of a friend on your Facebook page and commented on it? Facebook thanks you!
• Deep learning, as always, requires a profusion of training examples.
• Upon purchase of a Tesla vehicle, must agree to a data-sharing policy with the company.
• Requiring so much data is a major limitation of deep-learning today. Yoshua Bengio, another high-profile AI researcher, agrees: "We can't realistically label everything in the world and meticulously explain every last detail to the computer. "
• The term unsupervised learning refers to a broad group of methods for learning categories or actions without labelled data.
• In machine-learning jargon, Will's network "overfitted" to its specific training set.
• They are overfitting to their training data and learning something different from what we are trying to teach them. 
• Commercial face-recognition systems tend to be more accurate on white male faces than on female or non white faces. Camera software for face detection is sometimes prone to missing faces with dark skin and to classifying Asian faces as "blinking".
• The spread of real-world AI systems trained on biased data can magnify these biases and do real damage.
• Should the data sets being used to train AI accurately mirror our own biased society - as they often do now - or should they be tinkered with specifically to achieve social reform aims? And who should be allowed to specify the aims or do the tinkering.
• More generally, you can often trust that people know what they are doing if they can explain to you how they arrived at an answer or a decision.
• The dark secret at the heart of AI.
• Ian Goodfellow, an AI expert who is part of the Google Brain team, says, "Almost anything bad you can think of doing to a machine-learning model can be done right now...and defending it is really, really hard".
• It's misleading to say that deep networks "learn on their own" or that their training is "similar to human learning". Recognition of the success of these networks must be tempered with a realization that they can fail in unexpected ways because of overfitting to their trains data, long-tails effects and vulnerability to hacking.
• The formidable challenges of balancing the benefits of AI with the risks of its unreliability and misuse.

Chapter 7 On Trustworthy and Ethical AI

• tipsy: pompette
• sobering: qui donne à réfléchir
• menial: subalterne
• rickshaw: pousse-pousse
• to canvass: sonder l'opinion
• creepy: horreur
• hamstrung: couper les tendons d'Achille
• spur: voie secondaire
• staple: de base
• contrived: imaginé

• Facebook, for example, applies a face-recognition algorithm to every photo that is uploaded to its site, trying to detect the faces in the photo and to match them with known users (at least those users who haven't disabled this feature)
• Privacy is an obvious issue. Even if I'm not on Facebook (or any other social media platform with face recognition), photos including me might be tagged and later automatically recognized on the site, without my permission.
• "We deserve a world where we're not empowering governments to categorize, track and control citizens"
• My own opinion is that too much attention has been given to the risks from superintelligent AI and far too little to deep learning's lack of reliability and transparency and its vulnerability to attacks.

PART THREE Learning to Play

Chapter 8 Rewards for Robots

• preposterously: absurde
• nagging: tenace
• covertly: secrètement
• oblivious: inconscient
• wag: remuer
• puddle: flaque
• awash: inondé
• wry: ironique
• treat: friandise

• "Reward behavior like and ignore behavior I don't"
• Operant conditioning inspired an important machine-learning approach called reinforcement learning. Reinforcement learning contrasts with the supervised learning method.
• Reinforcement learning requires no labelled training examples. Instead, an agent - the learning program - performs actions in an environment (usually a computer simulation) and occasionally receives rewards from the environment.
• Reinforcement learning: learning too much at one time can be detrimental.
• Q-learning
• Exploration versus exploitation balance

Chapter 9 Game On

• stance: position
• paddle: pagaie
• to tally up: faire le total
• devious: sournois
• windfall: aubaine
• gobsmacked: estomaqué
• pruning: élagage

• In reinforcement learning we have no labels.
• Learning a guess from a better guess.
• Three all-important concepts: the game tree, the evaluation function and learning by self-play.
• AlphaGo acquired its abilities by reinforcement learning via self-play.
• The program chooses moves probabilistically.
• AlphaGo learns by playing against itself over many games.
• With its AlphaGo project, DeepMind demonstrated that one of AI's long-time grand challenges could be conquered by an inventive combination of reinforcement learning, convolutional neural networks and Monte Carlo tree search.

Chapter 10 Beyond Games

• imbuing: imprégner
• adversarial: conflictuel
• prowess: talent
• pesky: fichu

• Unlike supervised learning, reinforcement learning holds the promise of programs that can truly learn on their own, simply by performing actions in their "environment" and observe the outcome.
• How to think better: how to think logically, reason abstractly and plan strategically.
• Andrej Karpathy, Tesla's director of AI

PART FOUR Artificial Intelligence Meets Natural Language

Chapter 11 Words, and the Company They Keep

• to surmise: présumer
• beak: bec
• bent out of shape: upset and angry
• mind-boggling: impressionnant
• to impinge: empiéter
• wit: esprit
• alluring: séduisant 
• to seep through: suinter

• You shall know a word by the company it keeps.
• In linguistics, this idea is known more formally as distributional semantics.
• The semantic of words might actually require many dozens if not hundred of dimensions.
• It turns out that using word vectors as numerical inputs to represent words, as opposed to the simple one-hot scheme, greatly improves the performance of neural networks in NLP tasks.
• "word2vec": shorthand for "word to vector".
• You shall know a word by the company it keeps.
• The idea is to train the word2vec network to predict what words are likely to be paired with a given input word. Word vectors are also called word embeddings.
• Let's remember that the goal of this whole process is to find a numerical representation - a vector - for each word in the vocabulary, one that captures something of the semantics of the word.

Chapter 12 Translation as Encoding and Decoding

• mildew: moisissure, mildiou
• cringe: avoir un mouvement de recul
• spurred: incité
• jarring: qui secoue
• dazzled: éblouir
• wrongheaded: erroné
• gist: sens général

• Encoder, Meet Decoder
• Long short-term memory, LTSM units: the idea is that these units allow for more "short-term" memory that can last throughout the processing of the sentence.
• To measure the quality of a translation, BLEU essentially counts the number of matches - between words and phrases of varying lengths.
• My general experience is that the translation quality of, say, Google Translate declines significantly when it is given whole paragraphs instead of single sentences.
• While the skeletal meaning of this story comes through, subtle but important nuances get lost in all the translations.
• The main obstacle is this: like speech-recognition systems, machine-translation systems perform their task without actually understanding the text they are processing.
• "Machine translation...often involves problems of ambiguity that can only be resolved by achieving an actual understanding of the text - and bringing real-world knowledge to bear".
• The images were downloaded from repositories such as Flickr.com, and the captions for these images were produced by humans - namely, Amazon Mechanical Turk workers, who were hired by Google for this study.
• I'm certain that these systems will improve as researchers apply more data and new algorithms. However, I believe that the fundamental lack of understanding in caption-generating networks inevitably means that, as in language translation, these systems will remain untrustworthy.

Chapter 13 Ask Me Anything

• lodestar: guide
• puns: jeu de mots
• godsend: aubaine
• uncanny: mystérieux
• stunt: acrobatie
• parlour: petit salon
• dearth: manque
• bestowing: conférer
• to suss out: piger
• dubious: douteux
• to vie: concourir
• to forestall: prévenir
• inching: avancer doucement
• adversarial: conflictuel
• adversary: adversaire
• nefarious: abominable

• Overpromising and under-delivering are, of course, an all-too-common story in AI.
• The Winograd schemas are designed precisely to be easy for humans but tricky for computers.
• It seems to me to be extremely unlikely that machines could ever reach the level of humans on translation, reading comprehension and the like by learning exclusively from online data, with no real understanding of the language they process.
• Language also relies on commonsense knowledge of the other people with whom we communicate.

PART FIVE The Barrier of Meaning

Chapter 14 On Understanding

• endowed: doté
• insight: perspicacité
• bland: fade
• teeming: grouillant
• fraught: tendu
• what the heck? : c'est quoi ce bordel ?
• indulge: céder
• pun: jeu de mots
• yucky: dégoutant
• libel: diffamation
• feat: exploit

• Humans, in some deep and essential way, understand the situations they encounter, whereas no AI system yet possesses such understanding. While sate-of-the-art AI systems have nearly equalled (and in some cases surpassed) humans on certain narrowly defined tasks, these systems all lack grasp of the rich meanings humans bring to bear in perception, language and processing.
• Psychologists have coined a term - intuitive physics - for the basic knowledge and beliefs humans share about objects and how they behave. As very young children, we also develop intuitive biology: knowledge about how living things differ from inanimate objects.
• Because humans are a profoundly social species, from infancy on we additionally develop intuitive psychology: the ability to sense and predict the feelings, beliefs and goals of other people.
• Simulations appear central to the representation of meaning.
• For example, Lakoff and Johnson note that we talk about the abstract concept of time using terms that apply to the more concrete concept of money: You "spend" or "save" time. You often "don't have enough time".
• "I was given a warm welcome", "She gave me an icy stare", "He gave me the cold shoulder". Such phrasings are so ingrained that we don't realize we're speaking metaphorically. These metaphors reveal the physical basis of our understanding of concepts.
• Abstraction and analogy.
• Analogy-making in a very general sense as "the perception of a common essence between two things".
• "Without concepts there can be no thought, and without analogies there can be no concepts"
• Everyone in AI research agrees that core commonsense knowledge and the capacity for sophisticated abstraction and analogy are among the missing links required for future progress in AI.

Chapter 15 Knowledge, Abstraction and Analogy in Artificial Intelligence

• elusive: insaisissable
• rut: ornière
• commonsense: du bon sens; sensé; raisonnable
• imbue: imprégner
• mind-boggling: impressionnant
• grappling: s'agripper

• Lenat concluded that rule progress in AI would require machines to have common sense.
• Unwritten knowledge that humans have.
• Our commonsense knowledge is governed by abstraction and analogy.
• AI research often uses so-called microwords - idealized domains, such as Bongard problems, in which a researcher can develop ideas before testing them in more complex domains.
• Conceptual slippage, an idea at the heart of analogy-making.
• The concept of website slipped to the concept of wall, and the concept of writing a blog slipped to the concept of spray-painting graffiti.
• Have you ever struggled unsuccessfully to solve a problem, finally recognizing that you have been repeating the same unproductive thought process? This happens to me all the time; however, once I recognize this pattern, I can sometimes break out of the rut.
• "We Are really, Really Far Away"
• The modern age of artificial intelligence is dominated by deep learning, with its triumvirate of deep neural networks, big data and ultrafast computers.
• A small segment of the AI community has consistently argued for the so-called embodiment hypothesis: the premise that a machine cannot attain human-level intelligence without having some kind of body that interacts with the world.

Chapter 16 Questions, Answers and Speculations

• jaywalk: traverser en dehors des clous
• dart across: foncer
• inconspicuous: qui passe inaperçu
• pesky: fichu
• oxymoron: réalité paradoxale; exemple "mort-vivant"
• bewildering: déroutant
• beset: frappé
• ballparked: approximatif
• to elude: échapper
• witty: plein d'esprit
• elusive: insaisissable
• vexing: épineux
• foibles: manies
• addle: embrouillé
• headlong: tête la première

• The sort of core intuitive knowledge : intuitive physics, biology and especially psychology.
• It's worth remembering the maxim that the first 90 percent of a complex technology project takes 10 percent of the time and the last 10 per cent takes 90 per cent of the time.
• I believe that it is possible, in principle, for a computer to be creative.
• I've seen numerous computer-generated artworks that I consider beautiful.
• The creativity results from the teamwork of human and computer: the computer generates initial artworks and then successive variations, and the human provides judgment of the resulting works which comes from the human's understanding of abstract artistic concepts.
• "Human intelligence is a marvelous, subtle, and poorly understood phenomenon. There is no danger of duplicating it anytime soon"
• "Prediction is hard, especially about the future"
• The annoying limitations of humans, such as our slowness of thought and learning, our irrationality and cognitive biases, our susceptibility to boredom, our need for sleep and our emotions, all of which get in the way of productive thinking.
• Above all, the take-home message from this book is that we humans tend to overestimate AI advances and underestimate the complexity of our own intelligence.
• AI systems are brittle; that is, they make errors when their input varies too much from the examples on which they've been trained.
• We tend to anthropomorphize AI systems: we impute human qualities to them and end up overestimating the extent to which these systems can actually be fully trusted.

Gödel Ester Bach - Douglas Hofstadter

 

Chapitre X Niveaux de description et ordinateurs

• Le algorithmes, des descriptions exactes de processus dont ils désiraient l'exécution.
• Vers 1950, on a réussi à écrire des programmes appelés compilateurs, dont la fonction était de traduire des langages de compilation en langage machine.
• Un interpréteur est donc au compilateur ce qu'un interprète est à un traducteur.
• Un des secrets de la compréhension et de la création de l'intelligence réside dans le développement et le perfectionnement constants des langages capables de décrire des processus manipulation des symboles.
• Il suffit de prendre la plus exacte des sciences, la physique.
• Dans la plupart des systèmes que nous connaissons, les parties conservent leur identité en dépit des interactions, ce qui fait que nous continuons à voir les parties au sein du système.
• Phénomène de la supraconductivité : un flux d'électrons sans résistance au sein de certains solides, à des températures extrêmement basses.

...et fourmugue

• "Faut-il comprendre de façon holistique ou de façon réductionniste ?"
• Une fugue de fourmis, une "fourmugue".
• Les colonies de fourmis sont semblables aux cerveaux par de nombreux aspects.
• Vous étiez peut-être tellement captivé par ce que vous disiez que vous étiez parfaitement inconscient du reste.

Chapitre XI Cerveaux et pensées

• Ça n'est donc pas une si mauvaise idée, cette analogie entre le cerveau et une fourmilière.
• Cette idée de l'existence du général dans le particulier est d'une très grande importance.
• Les créatures fantastiques de notre logiciel cérébral qui naissent des étranges mélanges d'idées qui s'éveillent quand le corps s'endort.
• Il faut trouver une explication des activations de haut niveau des symboles qui ne s'appuie pas sur les phénomènes neuraux de bas niveau. Si c'est possible (et c'est une supposition essentielle des travaux actuels en intelligence artificielle) alors le cerveau ne serait pas l'unique support matériel de l'intelligence.
• Si, par contre, il n'existe aucun moyen d'effectuer des séquences d'activation de symboles sans avoir tout le matériel neutronique (que ces neurones soient réels ou simulés), c'est que l'intelligence est en fait un phénomène propre aux cerveaux et beaucoup plus difficile à comprendre qu'un phénomène devant son existence à une hiérarchie de lois de niveaux différents.
• Aux chapitres XVIII et XIX qui traitent de l'intelligence artificielle.

Les pensées édifiantes d'un fumeur de tabac

• Je feuilletais un curieux livre plein d'étranges dialogues sur de nombreux sujets, dont la biologie moléculaire, les fugues, le bouddhisme zen, et je ne sais quoi d'autre.
• Les ribosomes sont apparemment des genres d'objets sous-cellulaires qui prennent un message d'une forme donnée et le transforment en un message d'une autre forme.
• 
  
  
  SDS est une ribosomopathie (maladie due à un défaut de fabrication des ribosomes).
  
  
• Le mot "ceci" renvoie au tableau, et non pas à la pipe.

Chapitre XII Esprits et pensées

• Les caractéristiques communes à tous les papillons.
• Votre seule base de recherche, c'est votre tête. Vous n'avez accès à aucune information susceptible de vous aider pendant la durée de votre travail.
• Quelqu'un pourrait, par exemple, ignorer ce qu'est un éléphant, ou que la terre est ronde. Il est probable que dans tels cas, le réseau de symboles de cette personne et le vôtre seront si différents que vous ne pourrez pas établir une communication très importante.
• La culture affecte la pensée, il ne faut pas surévaluer le rôle de la langue dans le façonnage de la pensée.
• La culture russe est étrangère aux francophones.
• Je me suis parfois rendu compte que deux mélodies différentes traversaient mon esprit, rivalisant pour attirer "mon" attention.
• Lucas, "Minds, Machines, and Gödel" : ses opinions sont parfaitement à l'opposé des miennes et pourtant, pour se les former, il a mélangé beaucoup des mêmes ingrédients.

Chapitre XVI Auto-réf et auto-rep

• Le mécanisme qui lit les chaînes et produit les enzymes codées à l'intérieur de celles-ci s'appelle un ribosome.
• Les quatre types de base que l'on trouve dans les nucléotides de l'ADN sont :
• A : Adénine
• G : Guanine
• C : Cytosine
• T : Thymine
• Les enzymes, qui sont les instigatrices de presque tous les processus de vie, sont fabriquées par les ribosomes qui se trouvent dans le cytoplasme.
• L'ARN messager ou ARN-m sert à transporter les informations, c'est à dire le message génétique, des chambres moléculaires de l'ADN, jusqu'aux ribosomes flottant dans le cytoplasme.
• Le processus de copie de l'ADN dans l'ARN-m à l'intérieur du noyau s'appelle la transcription.
• Les enzymes appartiennent à la catégorie générale de molécules biologiques appelées protéines, et les ribosomes ont pour tâche de produire toutes les protéines, et pas seulement les enzymes.
• Une protéine est donc produite aminoacide par aminoacide par le ribosome.
• Pour que ces protéines potentielles soient extraites de l'ADN, il faut qu'il y ait non seulement des ribosomes, mais aussi des ARN polymérises, qui fabriquent l'ARN-m qui est apporté aux ribosomes.
• Ce n'est pas une coïncidence si les expressions "système d'autonomie suffisamment résistant" et "système formel suffisamment puissant" se ressemblent.
• En fait, en informatique, une certaine tendance à mélanger tous ces aspects apparemment distincts d'un système de traitement de l'information se dessine déjà nettement. C'est particulièrement vrai dans la recherche en intelligence artificielle, qui est généralement à l'avant-garde de la conception des langages informatiques.
• Quelle a été l'origine du code génétique et des mécanismes de décodage (ribosomes et molécules d'ARN de transfert) ?

Le Magnificrabe, en réalité

• On ne peut pas toujours comprendre sur quoi se fonde la beauté.

Chapitre XVII Church, Turing, Tarski et autres

• Nous en sommes maintenant au point où nous pouvons développer une des thèses centrales de ce livre, à savoir que tout aspect de la pensée peut être considérée comme une description de haut niveau d'un système qui, à une niveau inférieur, est géré par des règles simples, voire formelles.
• Un sens du jugement quant à ce qui est, ou n'est pas, important et nécessaire. Il va de pair avec un sens de la simplicité, un sens de la beauté. Mais d'où viennent ces intuitions ? Comment peuvent-elles surgir d'un système formel sous-jacent ?
• La thèse de Church-Turing est sans doute un des concepts les plus importants de la philosophie des mathématiques, des cerveaux et de la pensée.
• Srinavasa Ramajuan : sa mémoire, sa patience, et son pouvoir de calcul lui conféraient une capacité de généralisation, une intuition de la forme, et un pouvoir de modification rapide de ses hypothèses.
• Il semble que la "compréhension" exige au préalable l'assimilation et l'intégration d'une grande quantité d'informations.
• Si l'intelligence implique l'apprentissage, la créativité, des réactions émotionnelles, un sens de l'humour, un sens de la beauté, un sens de soi, alors il reste un long chemin à parcourir.
• La perception de la beauté est-elle un processus du cerveau ?
• Une conviction fausse dans le logiciel du cerveau est supportée par le matériel du cerveau qui, lui, fonctionne "parfaitement".
• La réalisation de la vraie intelligence nécessitera une poussée vers des niveaux toujours plus profonds, toujours plus proches des mécanismes cérébraux, si on veut que les machines atteignent les capacités humaines.
• A quoi réagissons-nous lorsque nous regardons un tableau et ressentons sa beauté ? Est-ce à la "forme" de ses lignes et points sur notre rétine ? C'est évidemment ça, puisque c'est sous cette forme-là que son contenu est transmis aux mécanismes d'analyse de nos cerveaux, mais la nature complexe du traitement effectué nous donne l'impression de ne pas être en train de regarder une simple surface à deux dimensions; nous réagissons à une sorte de signification interne du tableau, un aspect multidimensionnel piégé d'une manière ou d'une autre dans ces deux dimensions. C'est le mot "signification" qui compte ici. Nos cerveaux sont équipés d'interprètes qui reçoivent des formes à deux dimensions pour en "extraire" ensuite des concepts à plusieurs dimensions qui sont tellement complexes qu'ils échappent à toute description consciente. Nos réactions à la musique sont d'ailleurs du même type.
• Les qualités syntaxiques de de la forme et les aspects sémantiques de la forme.
• Au fur et à mesure que le temps passe le voile se lève de plus en plus sur la signification.
• Le paradoxe d'Épiménide : nature auto-référentielle.

Chapitre XVIII Intelligence artificielle : passé

• MIU : un alphabet, des axiomes, 4 règles de production. Théorème de Gödel : tu dois sortir du jeu et y réfléchir de l'extérieur pour le comprendre.
• Comment pouvez-vous savoir si ce que vous faites n'est pas parfaitement futile ?
• Cela ne signifie pas pour autant qu'il est impossible d'avoir une intuition de ce qui est ou n'est pas une voie prometteuse.
• On a un objectif global qui, localement, sert de guide. Une des techniques qui ont été mises au point pour convertir des objectifs globaux en stratégies locales de recherche de dérivations est appelée réduction des problèmes.
• Vous remarquerez que tout dépend de la façon de se représenter "l'espace du problème", c'est à dire de ce qui est perçu comme une réduction du problème (mouvement en avant vers l'objectif global), et de ce qui est perçu comme un grossissement du problème (mouvement éloignant de l'objectif).
• Et quand l'espace du problème est légèrement plus abstrait que l'espace physique, les humains ne savent souvent pas plus que faire que les chiens qui s'assoient et se mettent à aboyer.
• D'une certaine façon, tous les problèmes sont des versions abstraites du problème du chien et de l'os.
• Les solutions impliquant la restructuration de l'espace du problème se présentent le plus souvent à l'esprit comme un éclair de compréhension que comme le résultat d'une série de processus mentaux lents et délibérés. Ces éclairs d'intuition proviennent probablement du coeur même de l'intelligence, et leur source est, inutile de le dire, un secret jalousement protégé de notre cerveau.
• Ce qui fait, en tout cas, cruellement défaut à l'IA, ce sont des programmes capables de "prendre du recul" pour regarder ce qui se passe et, munis de ces informations, de se réorienter vers le but recherché.
• Y-a-t-il dans nos vies des situations très répétitives que nous appréhendons caque fois de façon tout aussi stupide parce que nous avons pas une vue assez globale pour percevoir leur similitude ? Voilà qui nous ramène à ce problème récurrent : "Qu'est-ce que la similitude ?" Nous le retrouverons en tant que thème de l'IA lorsque nous parlerons de la reconnaissance des formes.
• Il s'appuie sur un grand nombre d'aptitudes différentes, comme doit le faire l'intelligence en général : un vaste recueil de connaissances, la technique de réduction des problèmes, un grand nombre de méthodes heuristiques, plus quelques astuces spéciales.
• La conscience analogique, qui est un aspect crucial de l'intelligence humaine.
• Quand une personne oublie quelque chose, cela signifie vraisemblablement qu'un pointeur de haut niveau a été perdu, et non pas que des informations ont été effacées ou détruites. Cela montre qu'il est extrêmement important de prendre note de la façon dont vous stockez de nouvelles expériences, car vous ne pouvez pas savoir à l'avance dans quelles circonstances, ou sous quel angle, vous voudrez extraire des informations de votre cerveau.
• Un principe général : quelque chose devient ennuyeux non pas quand vous avez épuisé son répertoire de comportement, mais quand vous avez défini les limites de l'espace contenant son comportement.
• Pourquoi certaines musiques sont-elles beaucoup lus profondes et beaucoup plus belles que d'autres ? C'est parce que leur forme est expressive, en tout cas pour certaines régions étranges de notre inconscient.
• Il est extrêmement intéressant que, dans le langage naturel, la syntaxe et la sémantique soient étroitement enchevêtrés.
• Ce type de décision se présente constamment : combien de niveaux devrait avoir un système ? Quelle quantité et quel type d'"intelligence" devraient être placés à quel niveau ? Ce sont là quelques-uns des problèmes les plus ardus auxquels l'IA doit actuellement faire face.

Chapitre XIX Intelligence artificielle : avenir

• Je pense que les "quasi" situations et les situations hypothétiques inconsciemment forgées constituent l'une des plus riches sources potentielles de compréhension de l'organisation et de la classification des perceptions de l'être humain.
• Songez à quel point nos vies mentales seraient pauvres si nous n'étions pas doués de ce pouvoir créatif de nous glisser hors de la réalité pour plonger dans de douces suppositions ! Du point de vue de l'étude des processus de réflexion humains, ce glissement, ou déplacement, est très intéressant, car il se produit la plupart du temps inconsciemment, ce qui signifie que l'observation des déplacements et des non-déplacements est très révélatrice de ce qu'est l'inconscient.
• Une des principales fonctions du réseau de concepts est de permettre de modifier légèrement des idées fausses pour les transformer en variantes qui peuvent être correctes.
• Essayer les idées récentes qui ont marché.
• Malaphore : recombinaison d'idées.
• Question : Un ordinateur pensant pourra-t-il additionner rapidement ? Réflexion : Peut-être pas. "Je pense, donc je n'ai pas accès au niveau auquel je somme".
• La véritable intelligence dépend intimement d'une capacité d'appréhension globale de son environnement, c'est à dire d'une possibilité programmée de "sortir du système", tout au moins à peu près dans la mesure où nous avons, nous, cette possibilité. Or, une fois qu'un programme aura cette capacité, vous ne pourrez plus le retenir ; il aura dépassé le point critique, et il ne vous restera plus qu'à faire face aux conséquences de ce que vous aurez créé.
• Nous restons exactement les mêmes bien que des milliers de neurones meurent chaque jour.

ChapitreXX Boucles Étranges ou Hiérarchies Enchevêtrées

• Le pionnier de la cybernétique, Norbert Wiener. Science des communications et de la régulation des informations.
• Hiatus : manque de continuité, coupure.
• épistémologie : branche de la philosophie qui s'intéresse à la connaissance scientifique.
• immixtion : action de s'immiscer 

• Vous avez bien un sens des désirs qui découle du substrat physique de votre esprit.
• Une des questions centrales de ce livre serait "les mots et les pensées suivent-ils des règles formelles ? ". Un des grands objectifs de ce livre est de mettre en évidence les multiples niveaux de l'esprit/cerveau, et j'ai essayé de montrer pourquoi l'ultime réponse à cette question est : "oui, à condition de descendre au plus bas niveau, le matériel, pour trouver les règles".
• En fait les règles de fond ne changent pas. Les neurones agissent toujours selon le même processus simple.
• Vous avez accès à vos pensées, mais pas à vos neurones. Les règles logicielles de différents niveaux peuvent changer, mais les règles matérielles ne le peuvent pas et, en fait, c'est justement de leur rigidité que découle la souplesse du logiciel !
• Comment la conscience surgit de la jungle des neurones.
• Les gens ont un sens intuitif de la signification des faits parce que ils ont dans leurs cerveaux un matériel intégré qui incorpore des méthodes rudimentaires d'interprétation des faits.
• Mon idée là-dessus, c'est que le processus selon lequel nous décidons de ce qui est valide ou de ce qui est vrai est un art, et qu'il s'appuie aussi fermement sur un sens de la beauté et de la simplicité que sur des principes profondément enracinés de la logique, du raisonnement, ou de tout autre phénomène qui peut être objectivement formalisé.
• Théorème de Gödel : "La vérité est une chose trop sérieuse pour la confier entièrement aux théories mathématiques".
• La plus grande contradiction de nos vies, la plus difficile à affronter, est peut-être la connaissance que "il y a eu une époque où je n'étais pas vivant, et il y aura une époque où je ne serai pas vivant".
• La croyance plutôt orientale : "Le monde et moi ne sommes qu'un, et la notion de ma disparition est une contradiction intrinsèque". La croyance plutôt occidentale : "Je ne suis qu'une partie du monde, et je mourrai, mais après, le monde continuera sans moi"
• Principe d'incertitude d'Heisenberg : implication : non-déterminisme et rôle de l'observation.
• Magritte, La condition humaine : "C'est ainsi que nous voyons le monde : nous le voyons comme extérieur à nous, même si ce que nous éprouvons intérieurement n'en est qu'une représentation mentale"
• Libre arbitre : "Le système X fait-il des choix ?"
• Ce programme se surveille bien et a des idées sur ses idées, mais il n'est pas capable de surveiller tous les détails de ses processus et a donc une perception intuitive, et non pas une compréhension totale, de ses propres rouages. C'est cet équilibre entre la connaissance de soi et l'ignorance de soi que naît le libre arbitre.
• Peu importe que le système fonctionne de façon déterministe ou non ; nous le qualifions de "choisisseur" si nous pouvons nous identifier à une description de haut niveau du processus qui se déroule pendant l'exécution du programme. A un bas niveau (celui du langage machine), le programme ressemble à n'importe quel autre programme; à un haut niveau (réunitarisé), des qualités comme la "volonté", l'"intuition", la "créativité" et la "conscience" peuvent émerger.
• Escher a donc fait là une parabole graphique du Théorème d'incomplétude de Gödel. Voilà pourquoi les deux brins de Gödel et d'Escher sont si étroitement entremêlés dans mon livre.
• Quand on croit que l'on sait tout, il y a toujours plus.
• L'Offrande musicale est une fugue de fugues, une Hiérarchie Enchevêtrée comme celles d'Escher et de Gödel, une construction intellectuelle qui me rappelle, de façons inexprimables, la belle fugue à multiples voix qu'est l'esprit humain. Voilà pourquoi j'ai choisi, dans mon livre, de faire de Gödel, d'Escher et de Bach les trois brins d'une guirlande éternelle.

Ricercar à six voix

• BABBAGE : polymath, mathématicien, inventeur et cryptanalyste.
• Je me demande si vous accepteriez que j'essaie d'exécuter la tâche beaucoup moins grandiose consistant à multiplier MA PROPRE intelligence par six.

Data Preparation for Machine Learning - Part I, II, III - Jason Brownlee

 Preamble

• This post is an extract of the book "Data Preparation for Machine Learning" from Jason Brownlee.

Chapter 4: Data Preparation Without Data Leakage

• Naive application of data preparation methods to the whole dataset results in data leakage that causes incorrect estimates of model performance.
• Data preparation must be prepared on the training set only in order to avoid data leakage.
• Normalize data: scale input variables to the range 0-1
• Standardization estimates the mean and standard deviation values from the domain in order to scale the variables.
• The k-fold cross-validation procedure generally gives a more reliable estimate of model performance than a train-test split.

Part III: Data Cleaning

Chapter 5: Basic Data Cleaning

• Identify Columns that contain a Single Value
• Drop Rows
• Get duplicates
• Drop duplicates

Chapter 6: Outlier Identification and Removal

• Standard Deviation Method
• 1 standard deviation from the mean covers 68% of data
• 2 standard deviations from the mean covers 95 percent of data
• 3 standard deviations from the mean covers 99,7% of data
• Three standard deviations from the mean is a common cut-off in practice for identifying outliers in a Gaussian or Gaussian-like distribution.
• Interquartile Range Method
• Not all data is normal or normal enough to treat it as being drawn from a Gaussian distribution. A good statistic for summarizing a non-Gaussian distribution sample of data is the interquartile Range or IQR. The IQR is calculated as the difference between the 75th and 35th percentiles of the data and defines the box in a box and whisker plot.
• We can calculate the percentiles of a dataset using the percentile() NumPy function.
• Automatic Outlier Detection
• Each example is assigned a scoring of how isolated or how likely it is to be outliers based on the size of its local neighborhood.
• The scikit-learn library provides an implementation of this approach in the LocalOutlierFactor class.
• Standard deviation and interquartile range are used to identify and remove outliers from a data sample.

Chapter 7: How to Mark and Remove Missing Data

• Pandas provides the dropna() function that can be used to drop either columns or rows with missing data.

Chapter 8: How to Use Statistical Imputation

• Datasets may have missing values, and this can cause problems fro many machine learning algorithms. As such, it is good practice to identify and replace missing values for each column in your input data prior to modeling your prediction task. This is called missing data imputation, or imputing for short. A popular approach for data imputation is to calculate a statistical value for each column (such as a mean) and replace all missing values for that column with the statistic.
• Missing values must be marked with NaN values and can be replaced with statistical measures to calculate the column of values.
• The scikit-learn machine learning library provides the SimpleImputer class that supports statistical imputation.
• To correctly apply statistical missing data imputation and avoid data leakage, it is required that the statistics calculated for each column are calculated on the train dataset only, then applied to the train and test sets for each fold in the dataset. This can be achieved by creating a modeling pipeline where the first step is the statistical imputation, then the second step is the model. This can be achieved using the Pipeline class.
• Testing different imputed strategies. The author proposes to test different strategies when replacing data by comparing the mean, the median, the mode (most frequent) and constant(0) strategies. 

Box and Whisper plot of different imputation strategies

• We can see that the distribution of accuracy scores for the constant strategy may be better than the other strategies.

Chapter 9: How to Use KNN Imputation

• Although any one among a range of different models can be used to predict the missing values, the k-nearest neighbor (KNN) algorithm has proven to be generally effective, often referred to as nearest neighbor imputation.
• Missing values must be marked with NaN values and can be replaced with nearest neighbor estimated values.
• An effective approach to data imputing is to use a model to predict the missing values.
• One popular technique for imputation is a K-nearest neighbor model. A new sample is imputed by finding the samples in the training set "closest" to it and averages these nearby points to fill in the value.
• The use of a KNN model to predict or fill missing values is referred to as Nearest Neighbor Imputation or KNN imputation.
• The scikit-learn machine learning library provides the KNNImputer class that supports nearest neighbor imputation.
• The KNNImputer is a data transform that is first configured based on the method used to estimate the missing values.

Chapter 10: How to Use Iterative Imputation

• Missing values must be marked with NaN values and can be replaced with iteratively estimated values.
• It is common to identify missing values in a dataset and replace them with a numeric value. This is called data imputing, or missing data imputation. One approach to imputing missing values is to use an iterative imputation model.
• It is called iterative because the process is repeated multiple times, allowing ever improved estimates of missing values to be calculated as missing values across all features are estimated. This approach may be generally referred to as fully conditional specification (FCS) or multivariate imputation by chained equations (MICE).
• Different regression algorithms can be used to estimate the missing values for each feature, although linear methods are often used for simplicity.
• The scikit-learn machine learning library provides the IterativeImputer class that supports iterative imputation.

Conclusion

• In this first three parts of the Data preparation for Machine Learning Book, we review the main data cleaning possibilities.
• The following part will be related to "Feature Selection".

Better Deep Learning - Jason Brownlee - Models Weights Ensembles

 Preamble

• This blog post is an extract of the book "Better Deep Learning" from Jason Brownlee. 
• This blog post is related to "Better Predictions": Combine Model Parameters with average Model Weights Ensemble(Chapter 26)

Chenonceaux

Chapter 26: Combine Model Parameters with Average Model Weights Ensemble

• The model at the end of a training may not be stable or the best performing set of weights may not be usable as a final model. 
• One approach to address this problem is to use an average of the weights from multiple models seen toward the end of the training run. This is called Polyak-Ruppert averaging and can be further improved by using a linearly or exponentially decreasing weighted average of the model weights. In addition to resulting in a more stable model, the performance of the averaged model weights can also result in better performance.
• Learning the weights for a deep neural network model requires solving a high-dimensional non-convex optimization problem. A challenge with solving this optimization is that there are many good solutions and it is possible for the learning algorithm to bounce around and fail to settle in on one. In the area of stochastic optimization, this is referred to as problems with the convergence of the optimization algorithm on a solution, where a solution is defined by a set of specific weights values.
• "Polyak averaging consists of averaging together several points in the trajectory through parameter space visited by an optimization algorithm."
• "The basic idea is that the optimization algorithm may leap back and forth across a valley several times without ever visiting a point near of the bottom of the valley. The average of all the locations on either side should be close to the bottom of the valley though" - Deep Learning.
• The simplest implementation of Polyak-Ruppert averaging involves calculating the average of the weights of the models over the last few trains epochs.
• This can be improved by calculating a weighted average, where more weight is applied to more recent models, which is linearly decreased through prior epochs. An alternative and more widely approach used approach is to use an exponential decay in the weighted average.

Case Study

• Multiclass Classification problem

Plot of a multi class classification samples

• Multilayer Perceptron Model: in the problem suggested by the author, the training dataset is relatively small. There is a 10:1 ratio of examples in the training dataset to the holdout dataset. This mimics a situation where we may have a vast number of unlabeled examples and a small number of labeled examples with which to train a model. We will create 1100 data points. The problem will be trained for the first 100 points, 1000 will be held back in a test dataset, unavailable to the model.

Plot of Learning Curves of Accuracy on Train and Test Datasets

• The preliminary step before working on a Model Weight Ensemble consists at saving model weights to file during training, and later combine the weights from the saved models in order to make a final model.
• So, after this first step that establishes a baseline, the author suggest to create a new model from multiple existing models with the same architecture.
• Each model has a get_weights() function that returns a list of arrays, one for each layer in the model.
• After saving, loading and fitting with the weights the model, we are now ready to do some predictions.

Predictions with an Average Model Weight Ensemble

Plot of single Model test Performance and Model Weight Ensemble

Predictions with a Linear Weighted Average Ensemble

Accuracy of a linear Weighted Average Ensemble 

• I notice that the linear weight weighted average ensemble is not better the average model weighted ensemble. This is not what I was expected. The results in fact vary given the stochastic nature of the learning algorithm.

Predictions with an Exponentially Decreasing Average Ensemble

Accuracy of Single And Ensemble Model Weight Ensemble with an Exponential Decay

Conclusion

• Creating a model with the average of the weights from models observed towards the end of a training can result in a more stable and sometimes better-performing solution.
• This was the last chapter of "Better Deep Learning" book from Jason Brownlee. 
• I am very grateful to Jason Brownlee as this book helped me very much in my understanding of the tuning of neural networks.

Better Deep Learning - Jason Brownlee - Stacked Generalization Ensemble

 Preamble

• This blog post is an extract of the book "Better Deep Learning" from Jason Brownlee. 
• This blog post is related to "Better Predictions": how to combine predictions with Stacked Generalization Ensembles (chapter 25)

Alpine A310

Chapter 25: Learn to combine Predictions with Stacked Generalization

• Model averaging is an ensemble technique where multiple submodels contribute equally to a combined prediction. Model averaging can be improved by weighting the contributions of each submodel to the combined prediction by the expected performance of the submodel.This can be extended further by training an entirely new model to learn how to best combine the contributions from each submodel. This approach is called stacked generalization, or stacking for short, and can result in better predictive performance than any single contributing model.
• Stacked generalization is an ensemble method where a new model learns how to best combine the predictions from multiple existing models.

• Stacked generalization (or stacking) (Wolpert, 1992) is a different way of combining multiple models, that introduces the concept of a meta learner. Although an attractive idea, it is less widely used than bagging and boosting. Unlike bagging and boosting, stacking may be (and normally is) used to combine models of different types. The procedure is as follows:

1. Split the training set into two disjoint sets.
2. Train several base learners on the first part.
3. Test the base learnerson the second part.
4. Using the predictions from 3) as the inputs, and the correct responses as the outputs, train a higher level learner.
• Note that steps 1) to 3) are the same as cross-validation, but instead of using a winner-takes-all approach, we combine the base learners, possibly nonlinearly.

Case Study

• We are going to experiment the technique of "Stacked Generalization Ensembles" on a multi class classification problem. We will try to solve this classification problem with a classical MLP to establish a baseline, then in a second phase we will experiment the technique of Stacked Generalization Ensembles.

Plot of samples

• Then we run our MLP on the samples. The author suggests to create a sample of 1100 data points, the model being trained only on the first 100 points and the remaining 1000 points are held back in a test dataset.

Learning curves of Model Accuracy on Train and Test Dataset

First experimentation: a separate stack model 

• We start by training multiple submodels and saving them to files for later use in our stacking ensembles.
• We then train a meta-learner that will best combine the predictions from the submodels and we will see if the performance is better.
• The author prepare a training dataset for the meta-learner by providing examples from the test set to each of the submodels and collecting the predictions: dstack() and reshape() NumPy functions are used for combining arrays.
• The meta-learner is trained with a simple logistic regression algorithm from the scikit-learn library. The LogisticRegression class supports multi class classification (more than two classes) using a multinomial scheme.  
• Once fit the stacked model is used to make predictions on new data.

Fitting a logistic regression stacking model

• From the image above, the result of the experimentation is that the stacked model with a performance of 83,4 % outperforms all single model accuracy.

Second experimentation: an integrated stack model

• When using neural networks as submodels, it may be desirable to use a neural network as a meta-learner. Specifically, the sub-networks can be embedded in a larger multi-headed neural network that then learns how to best combine the predictions from each input submodel.
• The graph of the stacked model:

Stacked Generalization Ensemble of Neural Network Models

• The result of this stacked model is better that the separated one: 83,8% of accuracy compared to 83,4%.

Conclusion

• Stacked generalization is an ensemble method where a new model learns how to best combine the predictions from multiple existing models.
• We will experiment in the next chapter how to combine model parameters with average model weights ensemble.
• Big thanks to Jason Brownlee for helping me to understand these notions of Ensemble Learning.

Better Deep Learning - Jason Brownlee - Cyclic Learning Rate and Snapshot Ensembles

 Preamble

• This blog post is an extract of the book "Better Deep Learning" from Jason Brownlee. 
• This blog post is related to "Better Predictions": how to make better predictions using Cyclic Learning Rate and Snapshot Ensembles (chapter 24)
• I write this series of blog posts because it is a way for me to memorize by writing and also to experiment the pieces of code provides in the book.
• This book, as well as the other series of books from Jason Brownlee, are very helpful in my learning curve of Deep Learning. 
• The information related to neural network is complex and Jason Brownlee digests it for you in a didactic, pragmatic way and with concrete examples.

Voiture de sport

Chapter 24: Cyclic Learning rate and Snapshot Ensembles

• Model ensembles can achieve lower generalization error than single models but are challenging to develop with deep learning neural networks given the computational cost of training each single model.
• An alternative is to train multiple model snapshots during a single training run and combine their predictions to make an ensemble prediction. A limitation of this approach is that the saved models will be similar, resulting in similar predictions and predictions errors and not offering much benefit from combining their predictions.
• Effective ensembles require a diverse set of skillful ensemble members that have differing distribution of prediction errors. One approach to promoting a diversity of models saved during a single training run is to use an aggressive learning rate schedule that forces large changes in the model weights and, in turn, the nature of the model saved at each snapshot.
• Snapshot ensembles combine the predictions from multiple models saved during a single training run.
• Diversity in model snapshots can be achieved through the use of aggressively cycling the learning rate used during a single training run.
• One approach to ensemble learning for deep learning neural networks is to collect multiple models from a single training run.
• A key benefit of ensemble learning is in improved performance compared to the predictions from single models.
• A limitation of collecting multiple models during a single training run is that the models may be good, but too similar. This can be addressed by changing the learning algorithm for the deep neural network to force the exploration of different network weights during a single training run that will result, in turn, with models that have differing performance. One way that this can be achieved is by aggressively changing the learning rate used during training. 
• An approach to systematically and aggressively changing the learning rate during training to result in different network weights is referred to as Stochastic Gradient Descent with Warm Restarts or SGDR for short. This approach involves systematically changing the learning rate over training epochs, called cosine annealing.
• The cosine annealing (annealing = recuit in French) method has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before dramatically increased again. The model weights are subjected to the dramatic changes during training, having the effect of using "good weights" as the starting point for the subsequent learning rate cycle, but allowing the learning algorithm to converge to a different solution.
• "We let SGD converge M times to local minima along its optimization path. Each time the model converges, we save the weights and add the corresponding network to our ensemble. We then restart the optimization with a large learning rate to escape the current local minimum". In the original paper of the method, there is a nice graph that explains graphically the process:

Snapshot Ensembles: Train 1, get M for free, 2017

Case Study

• The first step is to establish a baseline for a multi class classification problem. We will then be able to compare the baseline with the Snapshot Ensembles.
• The multi class classification problem:

A multi class Classification problem

• The baseline for the MLP targeting to solve the multi class classification problem:

Model Accuracy on Train and Test Dataset over Each Training Epoch

• The coseline annealing schedule is an example of an aggressive learning rate schedule where learning rate schedule starts high and is dropped relatively rapidly to a minimum value near zero before being increased again to the maximum.
• So now the next step is to evaluate the cosine annealing learning schedule impact on the MLP:

MLP with Cosine Annealing Learning Schedule

• The final step is to evaluate the performance of the Snapshot Ensembles models. We will compare the ensemble vs the single models snapshot.

Snapshot Ensemble Performance

• End result of the experimentation: the snapshot ensemble performance achieved a performance accuracy of 81,9% vs a performance accuracy baseline of 81%.

Conclusion

• The snapshot ensemble performance is best described in the original paper of people who developed this snapshot ensemble model.
• Next step is to test how to Combine Predictions with Stacked Generalization Ensemble.