Motivation How to understand EM algorithm from a theoretical perspective? This post tries to understand EM as a form of alternative ascent of a lower bound of likelihood. The Key Trick of EM The key trick we need to remember is the usage of Jensen Inequality on logarithm. So we could swap Expectation and logarithm and obtain a lower bound on likelihood. Generally, we have such inequality, given a positive function $q(z)$ that sums to $1$ (probability density),
Motivation How to compute determinant or inversion of matrix with a low rank modificaiton? This is a very interesting and important math technique in statistical methods, since people frequently model covariance matrix or connectivity matrix as such: a matrix plus a low rank modification.
Motivation When you think about random walks, what shape do you think about? Is it like this? Or this? These are good examples of random walks in two or three dimensions. But what about random walks in higher dimensions?
TOC {:toc} Continuing Image Prior and Generative Model . Probabilistic Graphic Model comes into scene, when we want to model and deal with some complex distribution over many variables. When we start to add structure into the model, not everything depend on everything, then the dependency relationship among variables emerges as a graph structure.
看得见还是看不见, 这是一个问题! 如果问你, 你现在看到了什么, 你可以不假思索地说, 就是眼前的屏幕嘛. 如果稍微受过一点生理学或者神经科学训练的人也许会说, 我们看到的是这个充满电磁波的世界里, 光学波段辐射被眼球的光学元件折射之后, 落在我们视网膜上的影像. 然而我们不总是能看见落到我们视网膜上的东西. 稍加反思, 我们都会想到自己夏日午后看着窗外出神的时候, 其实并没有看见窗外有什么景色；对着讲台发呆的时候也没有看到黑板上的字是什么；对着击球手飞速击出的棒球同样略过了视网膜, 但是否产生了知觉只有他知道. 简而言之, 就像英语中see（看到）与look（看的动作）不同, 我们看到的东西绝不是落在我们视网膜上的光子所携带的所有信息1——你看不见不可见光也看不到偏振. 那么是什么决定了我们能看到什么以及看到的世界是什么样子呢? 我们可以拍脑门的说, 当我们用心去看就能看到(比如黑板或者窗外), 如果不用心, 那刺激就被我忽视(Neglect)了嘛——的确没错, 这就将我们引导到了视觉注意(Visual Attention)的领地. 不过这篇post中我们先不讨论注意的问题, 而要去关注一些与视觉意识相关的更基本的情况. 什么时候我们看不到 (When we do not see) 首先, 我们将自然环境以及落在视网膜上的光子一概称为物理刺激(Physical stimuli), 我们感觉到的那个世界称为主观感受(Perception)2. 那么刺激与感受不完全匹配的情况其实比比皆是, 下面举几个例子：
一些建立生活体系的基本原则1 知止而后有定，定而后能静，静而后能安，安而后能虑，虑而后能得 止者，所当止之地，即至善之所在也。知之，则志有定向。静，谓心不妄动。安，谓所处而安。虑，谓处事精详。得，谓得其所止。 四书章句集注 · 大学 回归简单的东西. 一本书 一张纸 一杯茶 一个程序. 一个时刻一心干一件事, 书桌桌面 电脑桌面东西不宜太多. 将复杂化为简单。(Cf. Physics from symmetry) 从感知(~mindfulness 正念 正觉)开始做起, (感知即仁, cf. 程子) Brain是进行任何思考活动的基础，在使用这一工具前，检查其状态，调谐(Tuning)其状态是必须的。感知与Meditation类似这种调谐身心的方法。（类似的使用自己的感知觉前 也应试图检查 反思其工具是否有问题，e.g. 是否眼镜片脏了） 从感知自己的状态，心情开始: 感知的目的是知识 了解, 进而是去改变, 调整, 不让不好的思维 习惯 状态控制自己的心智太久 一种改变自己状态的经典有效方法就是与自己写信/日记(which 自己曾经实践过很久)：通过文字经过屏幕的feedback，与自己交流理清思想。 自己常见的不适状态: 不知如何选择; 焦虑未来 焦虑已经做出的选择; 为(可能)惹别人不开心而焦虑; 为自己的self image受损而焦虑; 回避状态，因为一件事很困难而一直做别的, 不去处理这一件事，拖延 【face what your fear 性欲(Sexual Arousal)的annoying; 注意力不集中于一件事, 经常switch到手机 其他窗口中etc; 一段时间内拒绝思考未来, 不做计划; 拒绝回顾过去, 不去反思. 感知与反思过程所要弄清的一些问题: 目前的(心理)状态, motivation目标，外在压力来源(如ddl 事件), 扰动来源，环境状况 感知的基础是正常的Brain cognitive function. 作为生物体，保证足够的睡眠休息, 咖啡是必须的. (cf. 生生之道, 以促进生的方式对待生. )。 相反，不足够的睡眠之后就要去小心 risk of a malfunctioning brain ，比如如下方面： Memory 短时记忆力很差, 容易忘记事情 容易丢东西落东西; Attention 注意力很难集中; Self Perception 感知自己的状态、思想已经很困难，更难关注到，帮助到别人的状态； Planning 难以规划未来。 将理解 从自己推已及人 : 对自己有了充分的感知之后, 了解了自己的状态和情绪, 从而能更自信的交接事物, 理解他人，而不容易因为自己的情绪状态影响他人。 i.e. 关心自己是关心别人的发端和基础，知识基础以及习惯基础 人是有时间维度的. 完全生活在当下（一个没有过去也没有未来的点状人）也会带来负面结果. 在时间维度的一边是将来，我们需要 Expect and Plan，进行规划预期. 规划未来与感知当下 看似矛盾, 实际统一：没有规划的人会陷入永远的焦虑，因而难以感知当下；没有对当下自己状态、需要、目标的感知，也无从规划. (Duality of present and future) 为了做到一些事情(比如早睡)，就需要预先去规划/修改规划一番。 清楚的规划使人与别人的合作可以更方便一些，不会因为自己计划的不确定 而不断干扰他人计划。 【过于活在当下，不想即将发生的事儿，也许是自己不断迟到的一个原因 时间维度的另一端是过去，是memory，是experience，是dataset。 挖掘这一dataset，做pattern recognition, clustering (unsupervised learning), 是了解自己当下、了解自己状态的基础。 对于自己做的事引发的好或者不好的结果(these teaching signal enables supervised learning)，对于他人的exemplar，这些teaching dataset是Reinforcement Learning，改变自己的行为的基础。反思可以加强teaching signal，放大reward 与punishment，从而让人学的更快。(Cf. Skinner的鸽子) 不时整理数据集，写Note。(Cf. 曾子, 吾日三省吾身) 于了解自己发端，之后进行思想领域、知识领域、实践领域的探索 思想领域 不时检查思想领域的统一性 一贯性。思想/价值观本身像manifold 是由局部平凡的小块拼接起来的。每个局部上有统一的坐标，结构，整体上则非。 逻辑性并非人思维自带的模块，所以需要不时加入这一模块检查自身思想、价值体系 知识领域 整个的知识图景，与局部的细节。(Knowledge tree/network的重要性) 实践领域 Reinforcement Learning的重要性。 知行合一 Written in the last semester of undergrad. When I was intensively taking Math, Chinese Philosophy and History of Buddhism course. Trying to connect between Philosophical theory and the Psychological theory and Data Science to build a possible routine for life. Practise the idealism of breaking the barrier and let the different languages in different fields talk to each other. ↩︎
Antithesis 1 Search out necessary in everything for the sake of divine; and the divine we should search for the sake of a life of such happiness as our nature permits.
Motivation Given the popularity and power of diffusion models, the theoretical formulation of these models are not in unison. Because multiple groups have derived these models from different background, there exist multiple formulations, SDE, ODE, Markov Chain, Non-markov chain etc.
Hippo: Recurrent Memory with Optimal Polynomial Projection Motivation Hidden state in RNN represents a form of memory of the past. For a sequence, a natural way to represent the past sequence is to project it onto an orthonormal basis set. Here depending on the different emphasis of the past, we could define different measures on the time axis and define the basis set based on this measure. Then we can keep track of the projection coefficient on this basis when observing new data points.
[TOC] Motivation S4 sequence model is rising in the sequence modelling field. It dominates on long sequence modelling over RNN, LSTM and transformers. It’s both mathematically elegant and useful, and it’s trending, so why not write about it.
TOC {:toc} Motivation Consider a distribution $p(x)$, we could “convolve” it with a kernel $p(\tilde{x}\mid x)=q(\tilde{x}-x)$. The marginal distribution of $\tilde{x}$ is denoted as $p_\sigma(\tilde{x})$. We want to model the score of this convolved distribution and that of the original distribution $\nabla\log p_\sigma(\tilde{x})$ .
Motivation Julia is a very fast language. Julia hybrids many features from many sources. Here I’m taking note on its syntax and features by comparing to Python and Matlab. Basic Functions print is like in python BUT, no \n added at the end of each line. println has the \n String formatting is super easy, we can just use values of expressions. "$greet, $whom.\n" "1 + 2 = $(1 + 2)" Operators . operation extend its usage in matlab it means to vectorize sth. or broadcast one operation to all the element of the argument. This is super powerful, extending beyond arithmatics. You can broadcast everything by putting a . there. ... is like * in python, it could de-bundle a tuple or list into lots of variables. Types :: signify the type of sth. <: signify the type is a subtype of X. nothing equiv to None in python Similar to matlab true false not capitalized. Control Flow Control flow syntax is much like matlab if...elseif...end no indentation or :. Loop syntax is a bit like Python for .... in ..., with some caveat But if there are two things being looped, there needs to be a () , not like python. for (i,x) in enumerate(L) Functions Syntax is like
Motivation Last we we are pushed to a weird domain, using MTurk to assess perceptual properties of some images. And the major technical challenge is to make a form like question page in MTurk, using my close to zero prior knowledge on JavaScript. Here are something I learnt along this. In this note I’m introducing a few key examples of using JavaScript to interact with HTML.
MTurk MTurk is a online way to recruit subjects and perform tasks, widely used in psychology and machine learning to collect human perception and behavior data. Some terminology: HIT: Basically the task Requester: the experimenter Worker: the subjects An work through of a task (Image classification)
Motivation Writing a ML2 task with relatively complex timing dependency. https://monkeylogic.nimh.nih.gov/docs_CreatingTask.html#RuntimeVersion2 Adapter Chain child = Adapter1(tracker); parent = Adapter2(child); topmost = Adapter3(parent); scene = create_scene(topmost); run_scene(scene); This programming paradigm is really similar to that used in Tensorflow. The Adapters are like operator function on variables. But nothing really happen to the scene until run_scene
Note on Compiling Torch C Extensions Motivation Sometimes fusing operations in C library without using python can accelerate your model, especially for key operations that occurs a lot and lots of data pass through.
Environment Bug https://github.com/rosinality/stylegan2-pytorch/issues/70 Compiler not found bug We need to change compiler_bindir_search_path in ./stylegan2/dnnlib/tflib/custom_ops.pyNeed to be changed to have the C compiler on the machine. Note Visual Studio 2019 is not supported so have to use 2017!