50 items tagged
Motivation Given the popularity and power of diffusion models, the theoretical formulation of these models are not in unison. Because multiple groups have derived these models from different background, there exist multiple formulations, SDE, ODE, Markov Chain, Non-markov chain etc.
Hippo: Recurrent Memory with Optimal Polynomial Projection Motivation Hidden state in RNN represents a form of memory of the past. For a sequence, a natural way to represent the past sequence is to project it onto an orthonormal basis set. Here depending on the different emphasis of the past, we could define different measures on the time axis and define the basis set based on this measure. Then we can keep track of the projection coefficient on this basis when observing new data points.
[TOC] Motivation S4 sequence model is rising in the sequence modelling field. It dominates on long sequence modelling over RNN, LSTM and transformers. It’s both mathematically elegant and useful, and it’s trending, so why not write about it.
TOC {:toc} Motivation Consider a distribution $p(x)$, we could “convolve” it with a kernel $p(\tilde{x}\mid x)=q(\tilde{x}-x)$. The marginal distribution of $\tilde{x}$ is denoted as $p_\sigma(\tilde{x})$. We want to model the score of this convolved distribution and that of the original distribution $\nabla\log p_\sigma(\tilde{x})$ .
Motivation How to understand EM algorithm from a theoretical perspective? This post tries to understand EM as a form of alternative ascent of a lower bound of likelihood. The Key Trick of EM The key trick we need to remember is the usage of Jensen Inequality on logarithm. So we could swap Expectation and logarithm and obtain a lower bound on likelihood. Generally, we have such inequality, given a positive function $q(z)$ that sums to $1$ (probability density),
TOC {:toc} Motivation Recently, a line of research emerged in generative image models, diffusion models, which showed a competitive performance with GAN [^1]. More recently, a larger scale version of it gave rise to the ground breaking model DALL-E 2 and its precursor GLIDE.
TOC {:toc} Motivation Simply put, “kernel trick” is the finding that sometimes only inner product appears in the formulation of some algorithms. because of this, we could substitute the inner product with some fancier kernel function, i.e. inner product in some other spaces. This post is about another usage of kernel trick. Another usage is Kernel (ridge) Regression.
TOC {:toc} Motivation Understand the use of kernel in regression problems. For usage in unsupervised learning / dimension reduction, see notes on Kernel PCA. Kernel in Classification Kernel is usually introduced in SVM classification problems. The rationale is that a linearly non-separable dataset could be separable in a high-dimensional feature space using the mapping $\phi:\mathcal X\to\mathcal F$ .
Motivation There is a resurgent of interest in investigating and developing Hopfield network in recent years. This development is quite exciting in that it connect classic models in physics and machine learning to modern techniques like transformers.
Rationale Hopfield Network can be viewed an energy based model: deriving all properties from it. General RNN has many complex behaviors, but setting symmetric connections can prohibit it! No oscillation is possible in a symmetric matrix.
Motivation Word2Vec is a very famous method that I heard of since the freshman year in college (yeah it comes out in 2013). Recently, some reviewer reminds us of the similarity of the “analogy” learnt by the vector representation of words and the vector analogy of image space in GAN or VAE.
MTurk MTurk is a online way to recruit subjects and perform tasks, widely used in psychology and machine learning to collect human perception and behavior data. Some terminology: HIT: Basically the task Requester: the experimenter Worker: the subjects An work through of a task (Image classification)
Motivation Last we we are pushed to a weird domain, using MTurk to assess perceptual properties of some images. And the major technical challenge is to make a form like question page in MTurk, using my close to zero prior knowledge on JavaScript. Here are something I learnt along this. In this note I’m introducing a few key examples of using JavaScript to interact with HTML.
Motivation Writing a ML2 task with relatively complex timing dependency. https://monkeylogic.nimh.nih.gov/docs_CreatingTask.html#RuntimeVersion2 Adapter Chain child = Adapter1(tracker); parent = Adapter2(child); topmost = Adapter3(parent); scene = create_scene(topmost); run_scene(scene); This programming paradigm is really similar to that used in Tensorflow. The Adapters are like operator function on variables. But nothing really happen to the scene until run_scene
Problem Statement Given a bunch of noisy data, you want a smooth curve going through the cloud. As the points are noisy, there is no need to going through each point.
Note on Compiling Torch C Extensions Motivation Sometimes fusing operations in C library without using python can accelerate your model, especially for key operations that occurs a lot and lots of data pass through.
TOC {:toc} Philosophy The spirit of Variational Inference is to solve Bayesian inference problem with optimization. In the scenario of latent factor It’s not trying to use Bayes rule directly, but to fit this distribution within a class of distributions $q(z;\nu)$, by minimizing the KL-divergence between the 2 models.
Note on Gaussian Process Gaussian Process can be thought of as a Gaussian distribution in function space (or infinite dimension vector). One of its major usage is to tackle nonlinear regression problem and provide mean estimate and errorbar around it.
Note on Bayesian Optimization Related to Gaussain Process model Philosophy Bayesian Optimization applies to black box functions and it employs the active learning philosophy. Use Case and Limitation BO is preferred in such cases
Environment Bug https://github.com/rosinality/stylegan2-pytorch/issues/70 Compiler not found bug We need to change compiler_bindir_search_path in ./stylegan2/dnnlib/tflib/custom_ops.pyNeed to be changed to have the C compiler on the machine. Note Visual Studio 2019 is not supported so have to use 2017!
TOC {:toc} Motivation Sometimes the matrix (samples) to be correlated is too large, then you need to compute the correlation when the data is pouring in, i.e. online computing correlation.
Note on Laplacian-Beltrami (Diffusion) Operator Motivation Laplacian on graph and on discrete geometry (mesh) are very useful tools. One core intuition, just like Laplacian in $\R^n$ space, it’s related to diffusion and heat equation. Recall the diffusion equation is
Spectral Graph Theory and Segmentation Motivation Spectral Graph Theory is a powerful tool as it sits at the center of multiple representation. Connects to Graph and manifold, and linear algrbra. It’s related to dynamics on graph, related to Markov chain, random walk (diffusion.) Could be applied to any point cloud: images, meshes are suited. Could be used to perform clustering, segmentation etc. Linear Algebra Review There are several ways to see a eigenvalue problem
Note on Hyperbolic Geometry Reference Notes 2018 Lec Note 2015 Lecture note Ch5-3 Measurement in Hyperbolic Geometry [Cheatsheet / Note](http://home.iiserb.ac.in/~kashyap/MTH 520/lp.pdf) Motivation Hyperbolic geometry is a great source of inspiration for math art. Besides it is used to model some hierarchical data structure. Here I collected a few models
Motivation Major Reference Zeroth order optimization, or derivative free optimization is also known as the oracle problem. It’s nothing new to optimization community. Interest in ZOO algorithm resurges partly because it could be used in black box adversarial attack, if the softmax probability is given; and it could also be used in optimization of experimental output; and it could also be used for many design problem as the result has a non-analytical relationship with the parameters.
Krylov Subspace, Lancosz Iteration, QR and Conjugate Gradient Motivation In practise, many numerical algorithms include iteratively multiply a matrix, like power method and QR algorithm. All these algorithms have their core connected to a single construct, Krylov subspace and a operation, Lancosz Iteration. So this note motivates to understand this core.
TOC {:toc} Deep Learning Environment Currently we find that multiple version of CUDA could be installed on windows. And different frameworks could use different CUDA version nicely together. PyTorch Tensorflow Co-environment Currently, we can have
Objective Here I want to compare several common deep learning frameworks and make sense of their workflow. Core Logic Tensorflow General Comments: TF is more like a library, in which many low-level operations are defined and programs are long. In contrast, Keras which can use tensorflow as backend has the similar level of abstraction as PyTorch, which is a higher level deep learning package. TFLearn may also be a higher level wrapper.
TOC {:toc} Note on Online Regression Algorithm Least Square Problem Classical least square linear regression is $$ \hat \beta_{ls}=\arg\min_\beta\|y-X\beta\|^2_2 $$ With regularizations it becomes a ridge or lasso regression problem
Motivation Sometimes we want to examine the Hessian or Jacobian of a function w.r.t some variables. For that purpose, autogradient algorithm can help us. Autograd mechanism In Essence, Autograd requires a computational graph. (Directed Acyclic Graph) For each computational node (e.g. $z=f(x,y)$), we define a forward computation $(x,y)\mapsto z,\ z=f(x,y)$ mapping bottom to top, and a backward computation mapping the partial derivative to top to the partial derivative to bottom. $\partial_z\mapsto (\partial_x,\partial_y); (gx,gy)=g(gz;x,y)$ .
Motivation Unlike matlab, plotting dynamics in python is not as easy or straight forward to use. And to interact with the figure is not always as simple as matlab native plotting routines.
Computer Graphics environment in Matlab @(515.1 Computer Graphics)[matlab] Recently, we are using matlab to do computer vision experiments. Thus this note introduces some function controlling the elementary graphics environment. For Computer Graphics, the basic 3 components are
TOC {:toc} L-BFGS algorithm Motivation L-BFGS is one of the not so simple optimization algorithm that we may encounter in large scale optimization problems. Not so simple means it’s not simply a first order algorithm, and the deviation from that is well motivated by theoretical arguments. So this note target to understand this algorithm
Multi-platform Matlab compiling When trying to compile a matlab code, I faced and solved many errors either on the Windows platform and the MacOS platform. Here I note some prominent difference across platform and how to debug the compiling process.
Installation Official note on installation https://caffe.berkeleyvision.org/installation.html Installing CPU version on CHPC Install Miniconda Install caffe using condaconda install -c intel caffe lsb_release -d Description: CentOS release 6.10 (Final) Building GPU version on CHPC (Not succeeded yet…. aborted)
Some Computation on Sphere (Updating) Motivation Recently, in research, we encounter quite a few statistical problems on sphere. For example, Head direction tuning 3d direction of object 3d direction of body parts Some 3d tuning There are many standard statistical operations on Euclidean space, like getting mean, standard deviation and generate uniform distribution, fitting a model etc. We can perform these operation without thinking.
Note on Optimization on Manifold Manifold is locally similar to $\R^n$ flat space, but globally not. Manifold is locally homeomorphic to a Euclidean space of the same dimension. Besides there is Riemann logrithm map that connect the local vector space $T_p$ to the neighbourhood of $p$ on the manifold. Thus, many local optimization algorithm that work on flat $\R^n$ space can work the same on neighborhood of a manifold. The unique thing of working on a manifold is how to transport the local direction information in one neighborhood into another neighborhood.
Using Google Cloud Service for Large Scale Image Labelling Installing Google SDK https://cloud.google.com/sdk/docs/quickstart-windows New a Google Cloud Platform Project Download Google Cloud SDK After installation run gcloud init and log in to your account there! Select the GCP Project and the computing zone Finish the SDK configuration! Installing Google API for different programs (like Vision we use) https://cloud.google.com/python/
How to Parse Latin/Greek Anatomical/Physiological Terms TOC {:toc} 大部分现代医学 解剖术语似乎是文艺复兴之后生理学家创造的，但为了显得高端，使用当时的国际学术语言Latin书写。因为当时Latin在当时早已不是日常语言，所以大多是科学家们非常规整的用Classic Latin/Greek 的词根Root 词缀Pre-/Suffix 变位法Conjugation，人工构词产生的，算New Latin。所以知道了词根之后拼写起来单词就会像写中文组词一样，记忆起来也容易1，也更容易根据词形记忆解剖结构的位置。 词根 单个词大多是由词根词缀组合而成 比如 肋骨rib= costa costae(单数形容词) costarum(复数形容词) 腹股沟groin= Ilium Ilio(变位形式) 合在一起成了Iliocostalis 髂肋肌 Wikipedia List of medical roots, suffixes and prefixes
TOC {:toc} Python is a script language. Most of the packages are written in python, and they are just a bunch of script files in a folder hierachy. For these packages, it’s just as easy as find the script file and run them.
TOC {:toc} Objective Build the software environment for Scientific Computing Data Analysis and Deep Learning for a GPU enabled Linux work station. This post majorly summarizes the tools and references for building up a Linux Working Environment. I’ll update the errors and trouble shooting notes as I encounter them.
DeepLabCut Trouble Shooting @(Ponce Lab) TOC {:toc} Install DLC Windows machine, follow the steps in install tutorial to establish the whole conda environment in the machine. Fail at first step Many of us just fail at first step, some error message like
Motivation Although there are a millennium of methods for neural and behavioral signal recording, the questions asked about the neural data is ususally less diverse. Ultimately, everything is number and we process numbers with algorithm.
TOC {:toc} Problem Setting The original problem of non-negative matrix factorization is simple, if the dissimarity $D(A\|HW)$ between original matrix and reconstructed one is L2 distance than, $$ argmin_{H,W} \|A-HW\|_F^2, \\ s.t.\ W\succeq0, H\succeq0 $$The non-negative constraint applies element-wise.
TOC {:toc} Constrained CMA-ES Algorithm Target CMA-ES is originally used in unconstrained optimization. To adapt it into constrained optimization and we have to handle the boundary in some way. So how could it handle this geometric boundary?
TOC {:toc} 最近在阅读1，是以为记。 Objective of Algorithm 目标 Belief Propagation算法想解决的是Markov随机场，Bayes网络等图模型的边缘概率估计，以及求解最可能的状态的问题。 有许多名字称呼这一General的算法，如sum-product, max-product, min-sum, Message Passing等，属于更general的Message Passing算法范畴。 同时这一算法可以说是一种通用框架或者philosophy，因此在不同结构的模型中有许多著名的特例，这些具体算法也有各自的名字(如前向后向算法，Kalman Filter等等) 对于统计学习问题，通常会区分模型与算法，模型设定一些假设，抽象现实的某个方面，建立问题的结构；而算法求解问题(很多时候是转化为优化问题来求解)。在这个post中将要介绍的Belief Propagation算法，属于后者，但为了理解他，我们首先需要理解他对应的模型，即概率图模型。 Graphical Models： What relates graph to probability? 第一次接触概率图模型的人（像我）都会问，概率和图这两者有什么关系呢？ 我们知道图是一种直观的表征事物之间二元关系的方法通常由$(\mathcal V, \mathcal E)$定点和边组成。在概率图模型中，顶点通常代表随机变量，而边代表随机变量之间的关系。
Larval zebrafish is a tiny vertebrate, really suitable for circuit study, as its translucency. Gross Anatomy Data Zebrafish brain explorer @ Harold Burgess Lab Cell Type Data Comprehensive Identification and Spatial Mapping of Habenular Neuronal Types Using Single-Cell RNA-Seq Standard Brain
Note on Automatic 3D Instance Segmentation Pipeline In this note I try to summarize several recent works on Automatic 3D Instance Segmentation, with most direct application to saturated reconstruction of neural morphology in an imaging volume (mostly scanning Electral Microscopy, but seems it can be generalized into other imaging modality), which is one of the most important method of high-throughput connectomics1.
How to automatically analyze behavior video? DeepLabCut is a powerful tool to rapidly1 train a neural network (based on ResNet) to track keypoints on movement videos, esp. those of moving human or animals. Thus this is a game changing tool for all kind of behavior quantification for neuroscience and psychology researchers (can be applied to nearly any behavioral science topic, e.g. motor learning, motor control, facial expression, social interaction…). The workflow is relatively simple and it scarcely takes time after the network have been trained, and the video analysis can be done automatically. Because of this it’s really favorable to the reserchers doing long term ecological video recording.
Overview Objective: Build a online blog. Most preferably Posts can be written in Markdown, which is the format of most of my notes. Math formula (in Latex syntax) and equation rendering are Solution