DL Notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Top univercities courses:\n",
    "\n",
    "1. MIT 6.S094: Deep Learning for Self-Driving Cars: \n",
    "https://www.youtube.com/watch?v=1L0TKZQcUtA&list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf\n",
    "2. MIT 6.034 Artificial Intelligence, Fall 2010:\n",
    "https://www.youtube.com/watch?v=TjZBTDzGeGg&list=PLUl4u3cNGP63gFHB6xb-kVBiQHYe_4hSi\n",
    "3. Caltech Machine Learning Course - CS 156:\n",
    "https://www.youtube.com/watch?v=mbyG85GZ0PI&list=PLD63A284B7615313A\n",
    "4. Stanford cs231n Winter 2016 Convolutional Neural Networks for Visual Recognition:\n",
    "https://www.youtube.com/watch?v=g-PvXUjD6qg&list=PLlJy-eBtNFt6EuMxFYRiNRS07MCWN5UIA\n",
    "Notes: http://cs231n.github.io/\n",
    "5. Oxford Deep Learning for NLP at Oxford with Deep Mind 2017:\n",
    "https://www.youtube.com/watch?v=RP3tZFcC2e8&list=PL613dYIGMXoZBtZhbyiBqb0QtgK6oJbpm\n",
    "Code: https://github.com/oxford-cs-deepnlp-2017\n",
    "6. Awesome courses. Machine Learning.\n",
    "https://github.com/prakhar1989/awesome-courses/#machine-learning\n",
    "7. Yandex Academy. Machine Learning cource 2014:\n",
    "https://www.youtube.com/playlist?list=PLJOzdkh8T5kp99tGTEFjH_b9zqEQiiBtC\n",
    "8. Deep Learning Summer School, Montreal 2016: http://videolectures.net/deeplearning2016_montreal/\n",
    "9. Berkeley Artificial Intelligence CS 188 Spring 2016:\n",
    "https://www.youtube.com/watch?v=3aCn2-Slaoc&list=PLIeooNSdhQE5kRrB71yu5yP9BRCJCSbMt\n",
    "10. Stanford Deep Learning Tutorial: http://ufldl.stanford.edu/tutorial/\n",
    "11. Stanford CS224d: Deep Learning for NLP 2016:\n",
    "https://www.youtube.com/watch?v=kZteabVD8sU&list=PLCJlDcMjVoEdtem5GaohTC1o9HTTFtK7 course http://cs224d.stanford.edu/\n",
    "12. Andrew Ng's Deep learning course on Coursera: https://www.deeplearning.ai/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other useful courses:\n",
    "\n",
    "1. Udacity Intro to Parallel Programming: https://www.udacity.com/course/intro-to-parallel-programming--cs344\n",
    "2. TensorFlow: https://www.youtube.com/watch?v=wuo4JdG3SvU&list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ\n",
    "3. TensorFlow for Poets: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/#0\n",
    "4. Every single Machine Learning course on the internet, ranked by your reviews: https://medium.freecodecamp.org/every-single-machine-learning-course-on-the-internet-ranked-by-your-reviews-3c4a7b8026c0\n",
    "5. Data Science 101 (Getting started in NLP): Tokenization tutorial http://blog.kaggle.com/2017/08/25/data-science-101-getting-started-in-nlp-tokenization-tutorial/\n",
    "\n",
    "## Math\n",
    "1. Khan Academy Math course: https://www.khanacademy.org/math\n",
    "2. Khan Academy Linear algebra: https://www.khanacademy.org/math/linear-algebra\n",
    "3. Essence of linear algebra: https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab\n",
    "4. Essence of calculus: https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr\n",
    "5. Computational Linear Algebra for Coders: https://github.com/fastai/numerical-linear-algebra/blob/master/README.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other videos:\n",
    "\n",
    "1. Who is Afraid of Non-Convex Loss Functions? http://videolectures.net/eml07_lecun_wia/\n",
    "2. Tensorflow and deep learning - without a PhD by Martin Görner: https://www.youtube.com/watch?v=vq2nnJ4g6N0\n",
    "3. TensorFlow Tutorial #02 Convolutional Neural Network: https://www.youtube.com/watch?v=HMcx-zY8JSg\n",
    "4. Visualizing and Understanding Deep Neural Networks by Matt Zeiler: https://www.youtube.com/watch?v=ghEmQSxT6tw\n",
    "5. Live Q&A with the Deep Learning Foundations Team: https://www.youtube.com/watch?v=HgLplBRpRcs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Books:\n",
    "\n",
    "1. Deep Learning. Ian Goodfellow and Yoshua Bengio and Aaron Courville. MIT Press.\n",
    "http://www.deeplearningbook.org/\n",
    "2. Neural Networks and Deep Learning. Michael Nielsen.\n",
    "http://neuralnetworksanddeeplearning.com/\n",
    "3. Free Machine Learning eBooks:\n",
    "https://www.packtpub.com/books/content/free-machine-learning-ebooks/\n",
    "4. Grokking Deep Learning. Andrew W. Trask.\n",
    "https://www.goodreads.com/book/show/31565758-grokking-deep-learning\n",
    "5. Artificial Inteligence:\n",
    "https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/chapter1.html\n",
    "6. Bootstrapping Machine Learning. Louis Dorard. \n",
    "http://www.louisdorard.com/machine-learning-book/\n",
    "7. The Quest for Artificial Intelligence. Nils J. Nilsson. Cambridge University Press.\n",
    "https://www.amazon.com/Quest-Artificial-Intelligence-Nils-Nilsson/dp/0521122937/?tag=widgetsamazon-20\n",
    "Free PDF: http://ai.stanford.edu/~nilsson/QAI/qai.pdf\n",
    "8. Artificial Intelligence: Foundations of Computational Agents. David L. Poole. Cambridge University Press.\n",
    "https://www.amazon.com/Artificial-Intelligence-Foundations-Computational-Agents/dp/0521519004/?tag=widgetsamazon-20\n",
    "9. Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). Richard S. Sutton.\n",
    "https://www.amazon.com/Reinforcement-Learning-Introduction-Adaptive-Computation/dp/0262193981/?tag=widgetsamazon-20\n",
    "10. Machine Learning: An Algorithmic Perspective (Chapman & Hall/Crc Machine Learning & Pattern Recognition). Stephen Marsland. https://www.amazon.com/Machine-Learning-Algorithmic-Perspective-Recognition/dp/1420067184\n",
    "11. Machine Learning: A Probabilistic Perspective. Kevin P. Murphy. MIT Press.\n",
    "https://www.goodreads.com/book/show/15857489-machine-learning\n",
    "12. Gödel, Escher, Bach: An Eternal Golden Braid. Douglas R. Hofstadter.\n",
    "https://www.amazon.com/G%C3%B6del-Escher-Bach-Eternal-Golden/dp/0465026567 MIT course: https://ocw.mit.edu/high-school/humanities-and-social-sciences/godel-escher-bach/ course content: https://www.youtube.com/watch?v=5jFhq3Rj6DI\n",
    "13. Applied Predictive Modeling. Max Kuhn, Kjell Johnson https://www.amazon.com/dp/1461468485"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Audiobooks:\n",
    "1. Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots. John Markoff.\n",
    "https://www.amazon.com/Machines-Loving-Grace-Common-Between/dp/B00Y3VXZSG\n",
    "2. The Singularity Is Near: When Humans Transcend Biology. Ray Kurzweil.\n",
    "https://www.amazon.com/Singularity-Near-Humans-Transcend-Biology/dp/B004ZF18PW\n",
    "3. How to Create a Mind: The Secret of Human Thought Revealed. Ray Kurzweil.\n",
    "https://www.amazon.com/How-Create-Mind-Thought-Revealed-ebook/dp/B007V65UUG\n",
    "4. Philosophy of Mind: Brains, Consciousness, and Thinking Machines. The Great Courses, Professor Patrick Grim.\n",
    "https://www.amazon.com/Philosophy-Mind-Consciousness-Thinking-Machines/dp/B00DTO5LQC\n",
    "5. Superintelligence: Paths, Dangers, Strategies. Nick Bostrom.\n",
    "https://www.amazon.com/dp/B00LPMFE9Y/\n",
    "6. The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. Pedro Domingos.\n",
    "https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/B014X01SS0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cool services / tools:\n",
    "\n",
    "1. Google AI experiments: https://aiexperiments.withgoogle.com/\n",
    "2. Linear Classification Loss Visualization: http://vision.stanford.edu/teaching/cs231n/linear-classify-demo/\n",
    "3. Deep Learning for Java: https://deeplearning4j.org/index.html\n",
    "4. Eliminate engineering bottlenecks in Deep Learning: https://www.floydhub.com/\n",
    "5. Word2Vec by Google: https://code.google.com/archive/p/word2vec/\n",
    "6. Udacity Self Driving Car Simulator: https://github.com/udacity/self-driving-car-sim\n",
    "7. Scientific search https://www.wolframalpha.com/\n",
    "8. Plot functions online: https://www.desmos.com/\n",
    "9. Become a Data Scientist: https://www.dataquest.io/\n",
    "10. A growing collection of evolution simulation games: https://evolsim.itch.io/\n",
    "11. Yolo - realtime object detection: https://pjreddie.com/darknet/yolo/\n",
    "12. 3D Convolutional Network visualization: http://scs.ryerson.ca/~aharley/vis/conv/\n",
    "13. Maze Hover (Labirint): https://bl.ocks.org/mbostock/6ba3a15c9f08742b3a80\n",
    "14. OpenAI Gym: https://gym.openai.com/\n",
    "15. Weka 3: Data Mining Software in Java: http://www.cs.waikato.ac.nz/ml/weka/\n",
    "16. deeplearn.js a hardware-accelerated machine intelligence library for the web: https://pair-code.github.io/deeplearnjs/\n",
    "17. Netscope. A web-based tool for visualizing neural network architectures (or technically, any directed acyclic graph): https://ethereon.github.io/netscope/quickstart.html\n",
    "18. STMVis - Visual Analysis for Recurrent Neural Networks http://lstm.seas.harvard.edu/\n",
    "19. Beautiful thematic maps with ggplot2 (only) https://timogrossenbacher.ch/2016/12/beautiful-thematic-maps-with-ggplot2-only/\n",
    "20. Deep Visualization Toolbox https://github.com/yosinski/deep-visualization-toolbox\n",
    "21. Neural network visualization toolkit for keras https://github.com/raghakot/keras-vis\n",
    "\n",
    "## Style transfer\n",
    "1. http://www.pikazoapp.com/\n",
    "2. https://deepart.io/\n",
    "3. https://artisto.my.com/\n",
    "4. https://prisma-ai.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Competitions / Datasets:\n",
    "1. The biggest community: https://www.kaggle.com/\n",
    "2. Data science competitions to save the world: https://www.drivendata.org/\n",
    "3. DeepTraffic. 6.S094: Deep Learning for Self-Driving Cars:\n",
    "http://selfdrivingcars.mit.edu/deeptraffic/\n",
    "4. CIPHER WAR. After a century of failing to crack an ancient script, linguists turn to machines:\n",
    "http://www.theverge.com/2017/1/25/14371450/indus-valley-civilization-ancient-seals-symbols-language-algorithms-ai\n",
    "5. Google Cloud and YouTube-8M Challenge: Predict YouTube video tags for a chance to win up to $30K: https://cloud.google.com/blog/big-data/2017/02/google-cloud-and-youtube-8m-challenge-predict-youtube-video-tags-for-a-chance-to-win-up-to-30k\n",
    "6. THE MNIST DATABASE of handwritten digits: http://yann.lecun.com/exdb/mnist/\n",
    "7. Fueling the Gold Rush: The Greatest Public Datasets for AI\n",
    "https://medium.com/startup-grind/fueling-the-ai-gold-rush-7ae438505bc2#.lslws43jc\n",
    "8. Boston Housing dataset: https://archive.ics.uci.edu/ml/datasets/Housing\n",
    "9. Financial Datasets: https://www.quandl.com/ and https://www.quantopian.com/home\n",
    "10. The CIFAR-10 dataset: https://www.cs.toronto.edu/~kriz/cifar.html\n",
    "11. Machine learning repository: http://archive.ics.uci.edu/ml/datasets.html\n",
    "12. notMNIST: http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html\n",
    "13. Classification datasets results. Discover the current state of the art in objects classification:\n",
    "http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html\n",
    "14. CS543/ECE549 Assignment 4: Deep Convolutional Neural Networks:\n",
    "https://inclass.kaggle.com/c/computer-vision-cs543-ece549"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical guides:\n",
    "\n",
    "1. Deep Learning, Applied. Project #1. Food Classification with Deep Learning in Keras / Tensorflow:\n",
    "http://blog.stratospark.com/deep-learning-applied-food-classification-deep-learning-keras.html\n",
    "2. AI Tool Building. Accelerating Deep Learning with Multiprocess Image Augmentation in Keras:\n",
    "http://blog.stratospark.com/multiprocess-image-augmentation-keras.html#multiprocess-image-augmentation-keras\n",
    "3. Build custom TensorFlow: https://www.tensorflow.org/install/install_sources and https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md\n",
    "4. TensorFlow Examples: https://github.com/aymericdamien/TensorFlow-Examples\n",
    "5. Practical tutorials, videos and challenges to learn new technologies, libraries and techniques: https://cambridgespark.com/content\n",
    "6. Running the Udacity Deep Learning Foundations image classification project on floydhub.com:\n",
    "https://github.com/ludwiktrammer/deep-learning/tree/master/image-classification\n",
    "7. 37 причин, почему ваша нейросеть не работает: https://habrahabr.ru/post/334944/\n",
    "8. Building Trust in Machine Learning Models (using LIME in Python): https://www.analyticsvidhya.com/blog/2017/06/building-trust-in-machine-learning-models/\n",
    "9. How to train a neural network to code by itself? https://becominghuman.ai/how-to-train-a-neural-network-to-code-by-itself-a432e8a120df\n",
    "10. Building a Facial Recognition Pipeline with Deep Learning in Tensorflow https://hackernoon.com/building-a-facial-recognition-pipeline-with-deep-learning-in-tensorflow-66e7645015b8\n",
    "11. Launch a GPU-backed Google Compute Engine instance and setup Tensorflow, Keras and Jupyter https://hackernoon.com/launch-a-gpu-backed-google-compute-engine-instance-and-setup-tensorflow-keras-and-jupyter-902369ed5272"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Whitepapers:\n",
    "\n",
    "1. Awesome - Most Cited Deep Learning Papers: https://github.com/terryum/awesome-deep-learning-papers\n",
    "2. MIT Evolving Neural Networks through Augmenting Topologies:\n",
    "http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf\n",
    "3. Google's Rules of Machine Learning: Best Practices for ML Engineering (Martin Zinkevich):\n",
    "http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf\n",
    "4. Understanding Neural Networks Through Deep Visualization:\n",
    "http://yosinski.com/media/papers/Yosinski__2015__ICML_DL__Understanding_Neural_Networks_Through_Deep_Visualization__.pdf\n",
    "5. LIPNET: END-TO-END SENTENCE-LEVEL LIPREADING: http://www.oxml.co.uk/publications/2016-Assael_Shillingford_LipNet.pdf Youtube video: LipNet in Autonomous Vehicles | CES 2017: https://www.youtube.com/watch?v=YTkqA189pzQ\n",
    "6. DEEPCODER: LEARNING TO WRITE PROGRAMS: https://openreview.net/pdf?id=ByldLrqlx\n",
    "7. Learning long-term dependencies with gradient descent is difficult: http://www.dsi.unifi.it/~paolo/ps/tnn-94-gradient.pdf\n",
    "8. Dropout: A Simple Way to Prevent Neural Networks from Overfitting: https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf\n",
    "9. List of recent whitepapers:\n",
    "https://docs.google.com/spreadsheets/d/1MT8LRuEn3xdJ7j6_XQjRZwV_fRmrpjaWwZKGk2Dqm4k/edit#gid=0\n",
    "10. On the Origin of Deep Learning: https://arxiv.org/abs/1702.07800\n",
    "11. Visual Question Answering Using Attention based Models: https://nishantrai18.github.io/resources/cs676_report.pdf\n",
    "12. Deep Forest: Towards An Alternative to Deep Neural Networks: https://arxiv.org/abs/1702.08835\n",
    "13. A\\* Sampling: https://papers.nips.cc/paper/5449-a-sampling.pdf\n",
    "14. All you need is a good init: https://arxiv.org/abs/1511.06422\n",
    "15. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift:\n",
    "https://arxiv.org/abs/1502.03167\n",
    "16. Transfer Learning: ftp://ftp.cs.wisc.edu/machine-learning/shavlik-group/torrey.handbook09.pdf\n",
    "17. Efficient Online and Batch Learning Using Forward Backward Splitting:\n",
    "http://machinelearning.wustl.edu/mlpapers/paper_files/jmlr10_duchi09a.pdf\n",
    "18. Intriguing properties of neural networks: https://arxiv.org/abs/1312.6199 PDF: https://cs.nyu.edu/~zaremba/docs/understanding.pdf\n",
    "19. Deal or No Deal? End-to-End Learning for Negotiation Dialogues: https://arxiv.org/abs/1706.05125\n",
    "20. One Model To Learn Them All: https://arxiv.org/abs/1706.05137\n",
    "21. Deep Bilateral Learning for Real-Time Image Enhancement http://groups.csail.mit.edu/graphics/hdrnet/data/hdrnet.pdf\n",
    "22. Comparison of 13 classic ML algorithms on 165 datasets. Data-driven Advice for Applying Machine Learning to Bioinformatics Problems https://arxiv.org/abs/1708.05070"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional\n",
    "1. Visualizing and Understanding Convolutional Networks: http://www.matthewzeiler.com/pubs/arxive2013/eccv2014.pdf\n",
    "2. ImageNet Classification with Deep Convolutional Neural Networks: http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf\n",
    "3. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification: https://arxiv.org/pdf/1502.01852.pdf\n",
    "4. Optimization on Submanifolds of Convolution Kernels in CNNs: https://arxiv.org/abs/1610.07008#\n",
    "5. A Neural Algorithm of Artistic Style: https://arxiv.org/abs/1508.06576\n",
    "6. A Fully Convolutional Neural Network for Speech Enhancement: https://arxiv.org/abs/1609.07132\n",
    "7. Learning to Segment Object Candidates: https://arxiv.org/abs/1506.06204\n",
    "8. The 9 Deep Learning Papers You Need To Know About (Understanding CNNs Part 3):\n",
    "https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html\n",
    "9. You Only Look Once: Unified, Real-Time Object Detection: https://pjreddie.com/media/files/papers/yolo_1.pdf\n",
    "10. PixelNN: Example-based Image Synthesis http://www.cs.cmu.edu/~aayushb/pixelNN/\n",
    "11. BlitzNet: A Real-Time Deep Network for Scene Understanding http://thoth.inrialpes.fr/research/blitznet/\n",
    "\n",
    "## RNN\n",
    "1. Chinese Poetry Generation with Recurrent Neural Networks: http://www.aclweb.org/anthology/D14-1074\n",
    "2. Automated Crowdturfing Attacks and Defenses in Online Review Systems: https://arxiv.org/abs/1708.08151\n",
    "\n",
    "## Reinforncement Learning\n",
    "1. Reinforcement learning in the brain: http://www.princeton.edu/~yael/Publications/Niv2009.pdf\n",
    "\n",
    "## SVM\n",
    "1. Text Categorization with Support Vector Machines: Learning with Many Relevant Features:\n",
    "http://www.cs.cornell.edu/people/tj/publications/joachims_98a.pdf\n",
    "\n",
    "## Sentiment Analysis\n",
    "1. A comparison of Lexicon-based approaches for Sentiment Analysis of microblog posts:\n",
    "http://ceur-ws.org/Vol-1314/paper-06.pdf\n",
    "\n",
    "## Multimodal\n",
    "1. Multimodal Deep Learning: https://people.csail.mit.edu/khosla/papers/icml2011_ngiam.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Math:\n",
    "1. Linear algebra explained in four pages: http://www.souravsengupta.com/cds2016/lectures/Savov_Notes.pdf\n",
    "2. Calculus Cheat Sheet: http://tutorial.math.lamar.edu/pdf/Calculus_Cheat_Sheet_All.pdf\n",
    "3. Statistics cheatsheet: http://web.mit.edu/~csvoss/Public/usabo/stats_handout.pdf\n",
    "4. Math for Machine Learning: https://people.ucsc.edu/~praman1/static/pub/math-for-ml.pdf\n",
    "5. The Mathematics of Deep Learning:\n",
    "http://www.vision.jhu.edu/tutorials/ICCV15-Tutorial-Math-Deep-Learning-Intro-Rene-Joan.pdf\n",
    "6. The Mathematics of Machine Learning: http://datascience.ibm.com/blog/the-mathematics-of-machine-learning/\n",
    "7. Linear Algebra: http://www.deeplearningbook.org/contents/linear_algebra.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Various materials:\n",
    "\n",
    "1. The 10 Algorithms Machine Learning Engineers Need to Know: http://www.kdnuggets.com/2016/08/10-algorithms-machine-learning-engineers.html\n",
    "2. 10 Popular Java Machine Learning Tools & Libraries:\n",
    "http://www.datasciencecentral.com/profiles/blogs/10-popular-java-machine-learning-tools-libraries\n",
    "3. Deep Learning for Noobs: https://hackernoon.com/supervised-deep-learning-in-image-classification-for-noobs-part-1-9f831b6d430d#.liec70fzq\n",
    "4. Bag of Words Meets Bags of Popcorn: https://www.kaggle.com/c/word2vec-nlp-tutorial\n",
    "5. Softmax Regression:\n",
    "http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/\n",
    "6. Understanding the Bias-Variance Tradeoff: http://scott.fortmann-roe.com/docs/BiasVariance.html\n",
    "7. A Tour of Machine Learning Algorithms: http://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/\n",
    "8. TensorFlow in a Nutshell — Part One: Basics: http://camron.xyz/index.php/2016/08/22/in_a_nutshell_part_one/\n",
    "9. Advanced Natural Language Processing Tools for Bot Makers – LUIS, Wit.ai, Api.ai and others: https://stanfy.com/blog/advanced-natural-language-processing-tools-for-bot-makers/\n",
    "10. The Black Magic of Deep Learning - Tips and Tricks for the practitioner:\n",
    "https://nmarkou.blogspot.com.cy/2017/02/the-black-magic-of-deep-learning-tips.html\n",
    "11. Image-to-Image Translation in Tensorflow: http://affinelayer.com/pix2pix/index.html\n",
    "12. Image-to-Image Demo. Interactive Image Translation with pix2pix-tensorflow: https://affinelayer.com/pixsrv/\n",
    "13. Yes you should understand backprop: https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b#.gwed5ainp\n",
    "14. Blog notes of ML newbe: Getting Started with Machine Learning:\n",
    "http://www.theamazingfedex.com/2017/02/getting-started-with-machine-learning.html\n",
    "15. Attacking machine learning with adversarial examples: https://openai.com/blog/adversarial-example-research/\n",
    "16. ELI5 for deep learning. Simple explanations for complex concepts. ELI5 means Explain like I'm 5 years old. https://deep-learning-concepts.appspot.com/\n",
    "17. Preprocessing for Machine Learning with tf.Transform:\n",
    "https://research.googleblog.com/2017/02/preprocessing-for-machine-learning-with.html?m=1\n",
    "18. Neural Network Hyperparameters: http://colinraffel.com/wiki/neural_network_hyperparameters\n",
    "19. Deep Forest: Towards an Alternative to Deep Neural Networks: https://news.ycombinator.com/item?id=13773127\n",
    "20. Choosing the right estimator: http://scikit-learn.org/stable/tutorial/machine_learning_map/\n",
    "21. Deep Visual-Semantic Alignments for Generating Image Descriptions: http://cs.stanford.edu/people/karpathy/deepimagesent/\n",
    "22. TensorFlow Wide & Deep Learning Tutorial: https://www.tensorflow.org/tutorials/wide_and_deep\n",
    "23. Ensemble Learning. Better Predictions Through Diversity: http://abeautifulwww.com/EnsembleLearningETech.pdf\n",
    "24. ReLU Was Yesterday, Tomorrow Comes ELU: http://www.picalike.com/blog/2015/11/28/relu-was-yesterday-tomorrow-comes-elu/\n",
    "25. ResNets, HighwayNets, and DenseNets, Oh My!\n",
    "https://chatbotslife.com/resnets-highwaynets-and-densenets-oh-my-9bb15918ee32#.4lldkm8r8\n",
    "26. Logarithmic Loss: https://www.kaggle.com/wiki/LogarithmicLoss\n",
    "27. Hacker's guide to Neural Networks: http://karpathy.github.io/neuralnets/\n",
    "28. Infographic: Best Practices for Training Deep Learning Networks:\n",
    "https://medium.com/intuitionmachine/infographic-best-practices-in-training-deep-learning-networks-b8a3df1db53#.jsfnrszdv\n",
    "29. Understanding Neural Network Batch Training: A Tutorial:\n",
    "https://visualstudiomagazine.com/articles/2014/08/01/batch-training.aspx\n",
    "30. Back propagation with TensorFlow: http://blog.aloni.org/posts/backprop-with-tensorflow/\n",
    "31. Basic Concepts in Machine Learning: http://machinelearningmastery.com/basic-concepts-in-machine-learning/\n",
    "32. The Unreasonable Effectiveness of Recurrent Neural Networks:\n",
    "http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "33. Sense2vec with spaCy and Gensim: https://explosion.ai/blog/sense2vec-with-spacy\n",
    "34. Binary Classification Tutorial with the Keras Deep Learning Library: http://machinelearningmastery.com/binary-classification-tutorial-with-the-keras-deep-learning-library/\n",
    "35. The Neural Network Zoo: http://www.asimovinstitute.org/neural-network-zoo/\n",
    "36. How HBO’s Silicon Valley built “Not Hotdog” with mobile TensorFlow, Keras & React Native: https://hackernoon.com/how-hbos-silicon-valley-built-not-hotdog-with-mobile-tensorflow-keras-react-native-ef03260747f3\n",
    "37. AI Designers Find Inspiration in Rat Brains: http://spectrum.ieee.org/biomedical/imaging/ai-designers-find-inspiration-in-rat-brains\n",
    "38. The Dark Secret at the Heart of AI: https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/\n",
    "39. Достижения в глубоком обучении за последний год: https://habrahabr.ru/company/mailru/blog/338248/\n",
    "40. Google shows how easy it is for software to remove watermarks from photos https://www.theverge.com/2017/8/18/16162108/google-research-algorithm-watermark-removal-photo-protection\n",
    "41. Netflix's architecture. Distributed Time Travel for Feature Generation https://medium.com/netflix-techblog/distributed-time-travel-for-feature-generation-389cccdd3907\n",
    "42. A mindmap summarising Machine Learning concepts, from Data Analysis to Deep Learning. Machine Learning Mindmap / Cheatsheet https://github.com/dformoso/machine-learning-mindmap\n",
    "43. https://github.com/MrMimic/data-scientist-roadmap Toturial coming with \"data science roadmap\" graphe. http://nirvacana.com/thoughts/becoming-a-data-scientist/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q&A:\n",
    "1. What is batch size in neural network?\n",
    "http://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network\n",
    "2. multi-layer perceptron (MLP) architecture: criteria for choosing number of hidden layers and size of the hidden layer?\n",
    "http://stackoverflow.com/questions/10565868/multi-layer-perceptron-mlp-architecture-criteria-for-choosing-number-of-hidde\n",
    "3. How does transfer learning work? https://www.quora.com/How-does-transfer-learning-work\n",
    "4. How do I decide the number of nodes in a hidden layer of a neural network?\n",
    "https://www.quora.com/How-do-I-decide-the-number-of-nodes-in-a-hidden-layer-of-a-neural-network\n",
    "5. How does ELU activation function help convergence, and what's its advantages over ReLU or sigmoid or tanh function?\n",
    "https://www.quora.com/How-does-ELU-activation-function-help-convergence-and-whats-its-advantages-over-ReLU-or-sigmoid-or-tanh-function\n",
    "6. What's the difference between gradient descent and stochastic gradient descent?\n",
    "https://www.quora.com/Whats-the-difference-between-gradient-descent-and-stochastic-gradient-descent\n",
    "7. Tradeoff batch size vs. number of iterations to train a neural network:\n",
    "http://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network\n",
    "8. Why should weights of Neural Networks be initialized to random numbers?\n",
    "http://stackoverflow.com/questions/20027598/why-should-weights-of-neural-networks-be-initialized-to-random-numbers\n",
    "9. What are good initial weights in a neural network?\n",
    "https://www.quora.com/What-are-good-initial-weights-in-a-neural-network\n",
    "10. How do you initialize your neural network weights?\n",
    "https://www.reddit.com/r/MachineLearning/comments/2qsje7/how_do_you_initialize_your_neural_network_weights/\n",
    "11. Where do I call the BatchNormalization function in Keras?\n",
    "http://stackoverflow.com/questions/34716454/where-do-i-call-the-batchnormalization-function-in-keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional\n",
    "1. Convolutional Neural Networks (LeNet): http://deeplearning.net/tutorial/lenet.html\n",
    "2. Convolutional Neural Network: http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/\n",
    "3. A Beginner's Guide To Understanding Convolutional Neural Networks:\n",
    "https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/\n",
    "4. An Intuitive Explanation of Convolutional Neural Networks:\n",
    "https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/\n",
    "5. UNDERSTANDING CONVOLUTIONAL NEURAL NETWORKS FOR NLP:\n",
    "http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/\n",
    "6. http://neuralnetworksanddeeplearning.com/chap6.html\n",
    "7. Convolutional Neural Networks (CNNs): An Illustrated Explanation:\n",
    "http://xrds.acm.org/blog/2016/06/convolutional-neural-networks-cnns-illustrated-explanation/\n",
    "8. Convolutional Neural Networks (Andrew Gibiansky's blog):\n",
    "http://andrew.gibiansky.com/blog/machine-learning/convolutional-neural-networks/\n",
    "9. Machine Learning is Fun! Part 3: Deep Learning and Convolutional Neural Networks:\n",
    "https://medium.com/@ageitgey/machine-learning-is-fun-part-3-deep-learning-and-convolutional-neural-networks-f40359318721#.h6uqq3etp\n",
    "10. One by One [ 1 x 1 ] Convolution - counter-intuitively useful:\n",
    "http://iamaaditya.github.io/2016/03/one-by-one-convolution/\n",
    "11. Using convolutional neural nets to detect facial keypoints tutorial:\n",
    "http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/\n",
    "12. Picasso: A free open-source visualizer for Convolutional Neural Networks: https://medium.com/merantix/picasso-a-free-open-source-visualizer-for-cnns-d8ed3a35cfc5\n",
    "13. Convolutional Methods for Text: https://medium.com/@TalPerry/convolutional-methods-for-text-d5260fd5675f\n",
    "\n",
    "### Style transfer\n",
    "1. Convolutional neural networks for artistic style transfer:\n",
    "https://harishnarayanan.org/writing/artistic-style-transfer/\n",
    "2. Style transfer: https://ml4a.github.io/ml4a/style_transfer/\n",
    "3. Experiments with style transfer: http://genekogan.com/works/style-transfer/\n",
    "4. How do these \"neural network style transfer\" tools work? https://jvns.ca/blog/2017/02/12/neural-style/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recommender Systems\n",
    "1. Evaluating Recommender Systems - Explaining F-Score, Recall and Precision using Real Data Set from Apontador:\n",
    "http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html\n",
    "2. Applying Deep Learning to Collaborative Filtering: How Hulu builds its industry leading recommendation engine:\n",
    "http://tech.hulu.com/blog/2016/08/01/cfnade.html\n",
    "3. Co-occurrence Approach to an Item Based Recommender:\n",
    "https://blogs.msdn.microsoft.com/carlnol/2012/06/23/co-occurrence-approach-to-an-item-based-recommender/\n",
    "4. An Inside Look at the Components of a Recommendation Engine:\n",
    "https://www.mapr.com/blog/inside-look-at-components-of-recommendation-engine\n",
    "5. Item-to-Item Collaborative Filtering and Matrix Factorization:\n",
    "https://www.ics.uci.edu/~welling/teaching/CS77Bwinter12/presentations/course_Ricci/13-Item-to-Item-Matrix-CF.pdf\n",
    "6. Quick Guide to Build a Recommendation Engine in Python:\n",
    "https://www.analyticsvidhya.com/blog/2016/06/quick-guide-build-recommendation-engine-python/\n",
    "7. How to Build a Recommender System: http://blogs.gartner.com/martin-kihn/how-to-build-a-recommender-system-in-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis\n",
    "1. Deep Learning Sentiment One Character at a T-i-m-e:\n",
    "https://gab41.lab41.org/deep-learning-sentiment-one-character-at-a-t-i-m-e-6cd96e4f780d#.1d6tewmhs\n",
    "2. How is deep learning used in sentiment analysis?\n",
    "https://www.quora.com/How-is-deep-learning-used-in-sentiment-analysis\n",
    "3. Lexicon-Based Methods for Sentiment Analysis: https://www.aclweb.org/anthology/J/J11/J11-2001.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN\n",
    "1. Introduction to Recurrent Networks and Applications in EEG Analysis: http://www.cs.colostate.edu/~idfah/main/tutorials/eegrnn\n",
    "2. Recurrent Neural Networks in Tensorflow I: http://r2rt.com/recurrent-neural-networks-in-tensorflow-i.html\n",
    "3. Recurrent Neural Networks: https://www.tensorflow.org/tutorials/recurrent\n",
    "4. Lab 12-1 RNN Basics:\n",
    "https://docs.google.com/presentation/d/1UpZVnOvouIbXd0MAFBltSra5rRpsiJ-UyBUKGCrfYoo/edit#slide=id.g1ed9069b96_0_184\n",
    "5. Going Deeper Into Reinforcement Learning: Understanding Q-Learning and Linear Function Approximation: https://danieltakeshi.github.io/2016/10/31/going-deeper-into-reinforcement-learning-understanding-q-learning-and-linear-function-approximation/\n",
    "6. Deep Learning for NLP Best Practices: http://ruder.io/deep-learning-nlp-best-practices/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM\n",
    "1. Written Memories: Understanding, Deriving and Extending the LSTM:\n",
    "http://r2rt.com/written-memories-understanding-deriving-and-extending-the-lstm.html\n",
    "2. Understanding LSTM Networks: http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
    "3. LSTM Networks for Sentiment Analysis on Tweets:\n",
    "http://k8si.github.io/2016/01/28/lstm-networks-for-sentiment-analysis-on-tweets.html\n",
    "4. A noob’s guide to implementing RNN-LSTM using Tensorflow:\n",
    "http://monik.in/a-noobs-guide-to-implementing-rnn-lstm-using-tensorflow/\n",
    "5. LSTM Networks for Sentiment Analysis: http://deeplearning.net/tutorial/lstm.html\n",
    "6. A Beginner’s Guide to Recurrent Networks and LSTMs: https://deeplearning4j.org/lstm.html\n",
    "7. Time Series Prediction with LSTM Recurrent Neural Networks in Python with Keras:\n",
    "http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/\n",
    "8. Demystifying LSTM neural networks:\n",
    "https://blog.terminal.com/demistifying-long-short-term-memory-lstm-recurrent-neural-networks/\n",
    "9. Exploring LSTMs: http://blog.echen.me/2017/05/30/exploring-lstms/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GAN\n",
    "1. Deep adversarial learning is finally ready 🚀 and will radically change the game: https://medium.com/intuitionmachine/deep-adversarial-learning-is-finally-ready-and-will-radically-change-the-game-f0cfda7b91d3\n",
    "2. Work in progress: Portraits of Imaginary People: http://mtyka.github.io/machine/learning/2017/06/06/highres-gan-faces.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blogs\n",
    "1. Colah's blog: http://colah.github.io/\n",
    "2. Hulu Tech Blog: http://tech.hulu.com/\n",
    "3. Blog of Andrej Karpaty: http://karpathy.github.io/\n",
    "4. Fast AI blog: http://www.fast.ai/\n",
    "5. Darknet - Joseph Chet Redmon's blog: https://pjreddie.com/\n",
    "6. Xu Cui: http://www.alivelearn.net/\n",
    "7. Apple Machine Learning Journal: https://machinelearning.apple.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Communities:\n",
    "1. Reddit channel Machine Learning: https://www.reddit.com/r/MachineLearning/\n",
    "2. Reddit channel Deep Learning: https://www.reddit.com/r/deeplearning/\n",
    "3. Udacity deep learning nano degree Slack channel: https://nd101.slack.com/\n",
    "4. Siraj's Slack channel: https://programming-wizards.slack.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Youtube channels:\n",
    "1. sentdex: https://www.youtube.com/channel/UCfzlCWGWYyIQ0aLC5w48gBQ\n",
    "2. Welch Labs: https://www.youtube.com/channel/UConVfxXodg78Tzh5nNu85Ew\n",
    "3. Zafarullah Mahmood: https://www.youtube.com/channel/UCG0Qzlcp-p9bKIojn3OCzdA\n",
    "4. Jeremy Howard: https://www.youtube.com/channel/UCX7Y2qWriXpqocG97SFW2OQ\n",
    "5. Thales Sehn Körting: https://www.youtube.com/channel/UCSd_7rz5nzSnzUYbjaCXC5g\n",
    "6. Victor Lavrenko: https://www.youtube.com/channel/UCs7alOMRnxhzfKAJ4JjZ7Wg\n",
    "7. Siraj Raval: https://www.youtube.com/channel/UCWN3xxRkmTPmbKwht9FuE5A\n",
    "8. Computer Science Center: https://www.youtube.com/channel/UC0YHNueF-3Nh3uQT0P4YQZw\n",
    "9. Neural Networks - The Nature of Code: https://www.youtube.com/playlist?list=PLRqwX-V7Uu6aCibgK1PTWWu9by6XFdCfh addition to The Nature of Code book: http://natureofcode.com/book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Podcasts:\n",
    "\n",
    "1. Linear Digressions: http://lineardigressions.com/\n",
    "2. Data Skeptic: https://dataskeptic.com/\n",
    "3. Learning Machines 101: http://www.learningmachines101.com/\n",
    "4. Machine Learning Guide: http://ocdevel.com/podcasts/machine-learning\n",
    "5. The O'Reilly Data Show Podcast: https://www.oreilly.com/topics/oreilly-data-show-podcast\n",
    "6. Talking Machines: http://www.thetalkingmachines.com/\n",
    "7. Becoming a Data Scientist Podcast: http://www.becomingadatascientist.com/category/podcast/\n",
    "8. Partially Derivative: http://partiallyderivative.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tools and Documentation:\n",
    "\n",
    "1. Pandas: http://pandas.pydata.org/pandas-docs/stable/index.html\n",
    "2. Scikit Learn (Python): http://scikit-learn.org/stable/user_guide.html\n",
    "3. TFLearn: http://tflearn.org/\n",
    "4. Keras: https://keras.io/\n",
    "5. TensorFlow: https://www.tensorflow.org/api_docs/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hardware\n",
    "1. Build a super fast deep learning machine for under \\$1,000:\n",
    "https://www.oreilly.com/learning/build-a-super-fast-deep-learning-machine-for-under-1000\n",
    "2. Build a Deep Learning Rig for $800:\n",
    "https://medium.com/@ncondo/build-a-deep-learning-rig-for-800-4434e21a424f#.77h84ur5f\n",
    "3. Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# nd101 additional materials:\n",
    "\n",
    "1. Preparation for Udacity Foundations of Deep Learning Nanodegree:\n",
    "https://docs.google.com/spreadsheets/d/1NZtIxDWiJ_B0UKhIDUk-wTZAT3Fxfh-fGwcQKXg1bQU/edit#gid=0\n",
    "2. DL-ND LinkedIn Networking Sheet: https://docs.google.com/spreadsheets/d/1PWj19V9kb7A1V_Kagu29MWG41kvlXgNVLoSg7HN4P4s/edit#gid=0\n",
    "3. Deep Learning Changelog 2/10:\n",
    "https://docs.google.com/document/d/1xIpSbMTYpf5g6fQ6vmGDFa21iDibQwWQ_oxITyfs1rE/edit#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Slack nicknames:\n",
    "1. Mat Leonard: @mat\n",
    "2. Dhruv Parthasarathy (director of learning at Udacity): @dhruvp\n",
    "3. Siraj Raval: @siraj\n",
    "4. Andrew Trask: @trask"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}