33 items tagged
Motivation Given the popularity and power of diffusion models, the theoretical formulation of these models are not in unison. Because multiple groups have derived these models from different background, there exist multiple formulations, SDE, ODE, Markov Chain, Non-markov chain etc.
Hippo: Recurrent Memory with Optimal Polynomial Projection Motivation Hidden state in RNN represents a form of memory of the past. For a sequence, a natural way to represent the past sequence is to project it onto an orthonormal basis set. Here depending on the different emphasis of the past, we could define different measures on the time axis and define the basis set based on this measure. Then we can keep track of the projection coefficient on this basis when observing new data points.
[TOC] Motivation S4 sequence model is rising in the sequence modelling field. It dominates on long sequence modelling over RNN, LSTM and transformers. It’s both mathematically elegant and useful, and it’s trending, so why not write about it.
Motivation How to understand EM algorithm from a theoretical perspective? This post tries to understand EM as a form of alternative ascent of a lower bound of likelihood. The Key Trick of EM The key trick we need to remember is the usage of Jensen Inequality on logarithm. So we could swap Expectation and logarithm and obtain a lower bound on likelihood. Generally, we have such inequality, given a positive function $q(z)$ that sums to $1$ (probability density),
Motivation There is a resurgent of interest in investigating and developing Hopfield network in recent years. This development is quite exciting in that it connect classic models in physics and machine learning to modern techniques like transformers.
Rationale Hopfield Network can be viewed an energy based model: deriving all properties from it. General RNN has many complex behaviors, but setting symmetric connections can prohibit it! No oscillation is possible in a symmetric matrix.
Motivations Many CNN models have become the bread and butter in modern deep learning pipeline. Here I’m summarizing some famous CNN structure and their key innovations as I use them.
Motivation Word2Vec is a very famous method that I heard of since the freshman year in college (yeah it comes out in 2013). Recently, some reviewer reminds us of the similarity of the “analogy” learnt by the vector representation of words and the vector analogy of image space in GAN or VAE.
Problem Statement Given a bunch of noisy data, you want a smooth curve going through the cloud. As the points are noisy, there is no need to going through each point.
Note on Compiling Torch C Extensions Motivation Sometimes fusing operations in C library without using python can accelerate your model, especially for key operations that occurs a lot and lots of data pass through.
Note on Gaussian Process Gaussian Process can be thought of as a Gaussian distribution in function space (or infinite dimension vector). One of its major usage is to tackle nonlinear regression problem and provide mean estimate and errorbar around it.
Note on Bayesian Optimization Related to Gaussain Process model Philosophy Bayesian Optimization applies to black box functions and it employs the active learning philosophy. Use Case and Limitation BO is preferred in such cases
Environment Bug https://github.com/rosinality/stylegan2-pytorch/issues/70 Compiler not found bug We need to change compiler_bindir_search_path in ./stylegan2/dnnlib/tflib/custom_ops.pyNeed to be changed to have the C compiler on the machine. Note Visual Studio 2019 is not supported so have to use 2017!
Motivation This is a simple example. https://github.com/ProGamerGov/pytorch-old-tensorflow-models if pretrained: self.load_state_dict(torch.hub.load_state_dict_from_url(model_urls['inceptionv1'], progress=progress)) The official blog about how to use this is here. Hosting Weights The major challenge is to publish weight online. For that you need a public file hosting service, which Google Drive and OneDrive can do.
Note on MiniMax (Updating) Motivation This is a very traditional way of solving turn based game like chess or tic-tac-toc. It’s climax is Deep Blue AI in playing chess. Note, some people think about GAN training procedure as a min-max game between G and D, which is also interesting.
Reinforcement Learning deals with environment and rewards. Agents have a set of actions to interact with environment (state $s_i$), and the environment will be changed by these actions $a_j$, from time to time, there will be reward coming from environment!
Note on GAN Note with reference to the Youtube lecture series Hongyi Li. Architecture Developments Self Attention Used in Self-Attention GAN and BigGAN class Self_Attn(nn.Module): """ Self attention Layer""" def __init__(self,in_dim,activation): super(Self_Attn,self).__init__() self.chanel_in = in_dim self.activation = activation self.query_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) self.key_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) self.value_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim , kernel_size= 1) self.gamma = nn.Parameter(torch.zeros(1)) self.softmax = nn.Softmax(dim=-1) # def forward(self,x): """ inputs : x : input feature maps( B X C X W X H) returns : out : self attention value + input feature attention: B X N X N (N is Width*Height) """ m_batchsize,C,width ,height = x.size() proj_query = self.query_conv(x).view(m_batchsize,-1,width*height).permute(0,2,1) # B X CX(N) proj_key = self.key_conv(x).view(m_batchsize,-1,width*height) # B X C x (*W*H) energy = torch.bmm(proj_query,proj_key) # transpose check attention = self.softmax(energy) # BX (N) X (N) proj_value = self.value_conv(x).view(m_batchsize,-1,width*height) # B X C X N out = torch.bmm(proj_value,attention.permute(0,2,1) ) out = out.view(m_batchsize,C,width,height) out = self.gamma*out + x return out,attention Style GAN BigGAN Conditional GAN Text Conditioning Text is processed and combined with noise vector.
Understanding of Adversarial Attack Why we should care adversarial examples? Why they are surprising? Adversarial example and feature visualization are related intrinsically. Basic algorithm underlying both is to generate image that activate a unit most. For Adversarial attack, the point is to change the image in an in-perceptible way but change the network output dramatically. On the other hand the feature visualization is to generate a perceptually meaningful pattern that represent a natural image that the unit like.
TOC {:toc} Deep Learning Environment Currently we find that multiple version of CUDA could be installed on windows. And different frameworks could use different CUDA version nicely together. PyTorch Tensorflow Co-environment Currently, we can have
Objective Here I want to compare several common deep learning frameworks and make sense of their workflow. Core Logic Tensorflow General Comments: TF is more like a library, in which many low-level operations are defined and programs are long. In contrast, Keras which can use tensorflow as backend has the similar level of abstraction as PyTorch, which is a higher level deep learning package. TFLearn may also be a higher level wrapper.
Motivation Sometimes we want to examine the Hessian or Jacobian of a function w.r.t some variables. For that purpose, autogradient algorithm can help us. Autograd mechanism In Essence, Autograd requires a computational graph. (Directed Acyclic Graph) For each computational node (e.g. $z=f(x,y)$), we define a forward computation $(x,y)\mapsto z,\ z=f(x,y)$ mapping bottom to top, and a backward computation mapping the partial derivative to top to the partial derivative to bottom. $\partial_z\mapsto (\partial_x,\partial_y); (gx,gy)=g(gz;x,y)$ .
Installation Official note on installation https://caffe.berkeleyvision.org/installation.html Installing CPU version on CHPC Install Miniconda Install caffe using condaconda install -c intel caffe lsb_release -d Description: CentOS release 6.10 (Final) Building GPU version on CHPC (Not succeeded yet…. aborted)
Note on CNN Interpretability 2 major way of interpreting CNN Feature visualization: See what a hidden neuron is interested in Attribution: See what part of image activate a filter or detector Activation Atlas These works try to find a tool kits for visualizing DeepNN and building up a human-computer interface of DeepNN.
Note on Feature Visualization Motivation We want to understand what the hidden units “represent” What are they tuned to? What’s the favorite stimuli? Why should we find the most excitable stimuli? Resources DeepDream.ipynb Tensorflow
Deep Unsupervised Learning Lecture notes from Berkeley-cs294 https://sites.google.com/view/berkeley-cs294-158-sp19/home Lec 1 Category Generative Model Non-generative representation learning Motivation for Unsupervised Learning Application Generate/predict fancy samples Detect Anomaly / deviation from distribution Which human can do quite well without training Data Compression (because of predictability) Use the inner representation to do other tasks! Pre-training Type of Question Core Question: Modeling a Distribution
Using Google Cloud Service for Large Scale Image Labelling Installing Google SDK https://cloud.google.com/sdk/docs/quickstart-windows New a Google Cloud Platform Project Download Google Cloud SDK After installation run gcloud init and log in to your account there! Select the GCP Project and the computing zone Finish the SDK configuration! Installing Google API for different programs (like Vision we use) https://cloud.google.com/python/
TOC {:toc} Objective Build the software environment for Scientific Computing Data Analysis and Deep Learning for a GPU enabled Linux work station. This post majorly summarizes the tools and references for building up a Linux Working Environment. I’ll update the errors and trouble shooting notes as I encounter them.
DeepLabCut Trouble Shooting @(Ponce Lab) TOC {:toc} Install DLC Windows machine, follow the steps in install tutorial to establish the whole conda environment in the machine. Fail at first step Many of us just fail at first step, some error message like
Motivation Although there are a millennium of methods for neural and behavioral signal recording, the questions asked about the neural data is ususally less diverse. Ultimately, everything is number and we process numbers with algorithm.
Note on Automatic 3D Instance Segmentation Pipeline In this note I try to summarize several recent works on Automatic 3D Instance Segmentation, with most direct application to saturated reconstruction of neural morphology in an imaging volume (mostly scanning Electral Microscopy, but seems it can be generalized into other imaging modality), which is one of the most important method of high-throughput connectomics1.
How to automatically analyze behavior video? DeepLabCut is a powerful tool to rapidly1 train a neural network (based on ResNet) to track keypoints on movement videos, esp. those of moving human or animals. Thus this is a game changing tool for all kind of behavior quantification for neuroscience and psychology researchers (can be applied to nearly any behavioral science topic, e.g. motor learning, motor control, facial expression, social interaction…). The workflow is relatively simple and it scarcely takes time after the network have been trained, and the video analysis can be done automatically. Because of this it’s really favorable to the reserchers doing long term ecological video recording.
TOC {:toc} Task discription To find/generate the stimuli that evoked the strongest response of a neuron in a visual system is in essense an optimization problem. But the optimization task on hand has several unique features that are essential to the choice of optimization algorithms, for example,
题记 一直计划着博士期间定期写一点note把自己最近学到的有趣的, 美妙的东西记下来. 如果读得是一个数学物理的博士, 或者理论神经科学的博士, 那这种Note就像是在数学世界中的探险笔记, 可以叫This week’s finding, 大约就是这周看了什么书, 学会了什么数学, 玩儿了什么Model, 发现了什么trick或者math game，做了什么优美的图. (可以参见之前挖出来好些有意思东西的一个站点 This Week’s Finds in Mathematical Physics UCR一个数学物理教授坚持了十多年的每周数学笔记) 不过现实中, 我读的是Neuroscience的博士, 大概只能写learning写不了什么finding了. (而对于Brain一周时间也学不了什么新东西…) 因此, Note的内容就会更庞杂: 一部分是技术性的, 新学会的数学、统计方法、机器学习方法, 也许会有新的实验技术以及相关的物理原理; 另一部分是理念类的，也许有最近听seminar听到的神经或者心理的实验结果，也可能是相关的有趣的哲学讨论。我想我会逐渐发现哪些内容更适合分享, 以及哪些内容写下来对自己以及对读者更有帮助, 经过一段时间的磨合，这个post series应该能形成自己的风格。