Jekyll2022-05-17T14:09:16-07:00https://celis.github.io/feed.xmlAlejandro CelisData ScientistAlejandro Celisacelisalas@gmail.comhttps://celis.github.ioRelevant articles2019-11-18T00:00:00-08:002019-11-18T00:00:00-08:00https://celis.github.io/posts/2019/08/blog-post-relevant-articles<p>Recently, together with <a href="https://github.com/JoseEliel">Jose Eliel Camargo</a> I have been exploring a very nice and simple idea. When writing scientific articles, researchers put a lot of effort collecting relevant references and placing them within their text for different puposes: to give credit, to guide the reader to other points of view, to support some statement, etc. This means that looking for papers which tend to be cited close to each other in a collection of scientific articles should provide a good way to extract a group of similar or relevant articles.</p>
<p>With this in mind, we have extracted reference lists from <a href="https://labs.inspirehep.net">inspirehep</a> using the
<a href="https://github.com/celis/inspirehep_api_wrapper">inspirehep python wrapper</a>. Each reference list for us is just a list of inspire article ids. We then trained a Skip-Gram model using the <a href="https://radimrehurek.com/gensim/">gensim</a> library implementation. We end up with a dense representation in the space of inspirehep article ids, from where we can extract similar items using cosine similarity. Very simple!</p>
<p>Lets look at some of the results, I will start with one of my favourites:</p>
<p>I retrieve the three closest articles by cosine similarity to the following classic article:</p>
<ul>
<li><a href="https://labs.inspirehep.net/literature/74886">Regularization and Renormalization of Gauge Fields</a>
Gerard ‘t Hooft, M.J.G. Veltman</li>
</ul>
<p>I get the following results:</p>
<ul>
<li>
<p><a href="https://labs.inspirehep.net/literature/74882">A Method of Gauge Invariant Regularization</a>
J.F. Ashmore</p>
</li>
<li>
<p><a href="https://labs.inspirehep.net/literature/74881">Dimensional Renormalization: The Number of Dimensions as a Regularizing Parameter</a>
C.G. Bollini, J.J. Giambiagi</p>
</li>
<li>
<p><a href="https://labs.inspirehep.net/literature/74400">Lowest order divergent graphs in nu-dimensional space</a>
C.G. Bollini, J.J. Giambiagi</p>
</li>
</ul>
<p>These results are very good, as these articles developed simultaneously with the article by Gerard ‘t Hooft and M.J.G. Veltman the concept of dimensional regularization.</p>
<p>Lets look at another article, starting with</p>
<ul>
<li><a href="https://labs.inspirehep.net/literature/11883">Broken Symmetries and the Masses of Gauge Bosons</a>
Peter W. Higgs</li>
</ul>
<p>we predict the following three most similar articles</p>
<ul>
<li>
<p><a href="https://labs.inspirehep.net/literature/40440">Broken symmetries, massless particles and gauge fields</a>
Peter W. Higgs</p>
</li>
<li>
<p><a href="https://labs.inspirehep.net/literature/50073">Spontaneous Symmetry Breakdown without Massless Bosons</a>
Peter W. Higgs</p>
</li>
<li>
<p><a href="https://labs.inspirehep.net/literature/12291">Broken Symmetry and the Mass of Gauge Vector Mesons</a>
F. Englert, R. Brout</p>
</li>
</ul>
<p>Which again, looks quite good taking into account the Nobe Prize for Physics in connection with the Higgs boson discovery. We are very happy with the results obtained so far and continue working on the topic.</p>Alejandro Celisacelisalas@gmail.comhttps://celis.github.ioRecently, together with Jose Eliel Camargo I have been exploring a very nice and simple idea. When writing scientific articles, researchers put a lot of effort collecting relevant references and placing them within their text for different puposes: to give credit, to guide the reader to other points of view, to support some statement, etc. This means that looking for papers which tend to be cited close to each other in a collection of scientific articles should provide a good way to extract a group of similar or relevant articles.Text embeddings in hyperbolic space2019-08-10T00:00:00-07:002019-08-10T00:00:00-07:00https://celis.github.io/posts/2019/08/blog-post-hyperbolic-embeddings<p>Here I review the idea of representation learning in hyperbolic space following [1-6]. I will focus on the application of these methods towards the generation of word embeddings from natural language in an unsupervised manner. The standard algorithms for generating word embeddings, such as word2vec or GloVe, generate word representations in a multidimensional Euclidean space. These have proven to be extremely useful for so called downstream tasks (such as text classification, word similarity and name entity recognition) due to their ability to capture semantic and syntactic relations among words when trained on large text corpora.</p>
<p>Representation learning in hyperbolic space was studied initially in the context of graphs (see [1] and references therein), finding interesting results. In hyperbolic space, a circle circumference grows exponentially with the radius, making this space suitable to embed tree structures where the number of nodes increases dramatically as the depth of the tree increases. This allows to capture the graph complexity in relatively low-dimensional embedding spaces.</p>
<p>Deriving word embeddings in hyperbolic space from large corpora in an unsupervised manner is a natural follow-up. The main motivation for building word embeddings in hyperbolic space is that these embeddings might be able to capture better hierarchical relations present in language, such as hypernym-hyponym relationships, and therefore have the potential to perform better than traditional embeddings on certain tasks.</p>
<p>There are different realizations of hyperbolic space. The authors of [3-4] stress that gradient based optimization can be formulated more efficiently in the hyperboloid model. It therefore seems conventient to use the latter as our realization of the hyperbolic space. The n-dimensional hyperboloid model is defined by the points lying on the forward sheet of an hyperboloid in (n+1)-dimensional Minkowski space. Some works, including [1], have used the Poincaré ball representation of hyperbolic space. At the end, one can map points in a hyperboloid model to a corresponding Poincaré ball representation, so choosing one or the other is a matter of convenience.</p>
<p>To generate word embeddings in hyperbolic space, we need to define a loss function whose optimization would yield the desired word representations. Ref. [2] uses the loss function defined in [1], constructing a graph of word co-occurrrences along the way. Ref. [4] on the other hand generalizes the Skip-Gram loss function with negative sampling by considering inner products with the Minkowski metric. As the authors of [4] comment, the definition of the objective function for the purpose at hand deserves further research.</p>
<p>Overall, further work seems needed to fully assess the potential of hyperbolic embeddings in the context of word representations from large text corpora.</p>
<p>Implementations of the model presented in [1] can be found in <a href="https://radimrehurek.com/gensim/models/poincare.html">gensim</a>
and <a href="https://github.com/facebookresearch/poincare-embeddings">facebook research</a>. Implementation of the code used in [4] can be found in <a href="https://github.com/lateral/minkowski">minkowski</a>.</p>
<p><strong>References:</strong></p>
<p>[1] <em>“Poincaré Embeddings for Learning Hierarchical Representations”</em> from Maximilian Nickel and Douwe Kiela <a href="https://arxiv.org/abs/1705.08039">arxiv:1705.08039</a>.</p>
<p>[2] <em>“Embedding Text in Hyperbolic Spaces”</em> from Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai and George E. Dahl <a href="https://arxiv.org/abs/1806.04313">arxiv:1806.04313</a></p>
<p>[3] <em>“Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry”</em> from Maximilian Nickel and Douwe Kiela <a href="https://arxiv.org/abs/1806.03417">arxiv:1806.03417</a></p>
<p>[4] <em>“Skip-gram word embeddings in hyperbolic space”</em> from Matthias Leimeister and Benjamin J. Wilson <a href="https://arxiv.org/abs/1809.01498">arxiv:1809.01498</a></p>
<p>[5] <em>“Poincaré GloVe: Hyperbolic Word Embeddings”</em> from Alexandru Tifrea, Gary Bécigneul and Octavian-Eugen Ganea <a href="https://arxiv.org/abs/1810.06546">arxiv:1810.06546</a></p>
<p>[6] <em>“Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings”</em> from Matt Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela and Maximilian Nickel <a href="https://arxiv.org/abs/1902.00913">arxiv:1902.00913</a></p>Alejandro Celisacelisalas@gmail.comhttps://celis.github.ioHere I review the idea of representation learning in hyperbolic space following [1-6]. I will focus on the application of these methods towards the generation of word embeddings from natural language in an unsupervised manner. The standard algorithms for generating word embeddings, such as word2vec or GloVe, generate word representations in a multidimensional Euclidean space. These have proven to be extremely useful for so called downstream tasks (such as text classification, word similarity and name entity recognition) due to their ability to capture semantic and syntactic relations among words when trained on large text corpora.