A lot of people are using quanteda to analyze social media posts because it is very fast and flexible, but they sometimes face dramatic slow down due to memory swapping caused by insufficient sizes of RAM. quanteda requires the size of RAM to be 5 times larger than the data to analyze, but it can […]
Relaxing R version requirement
Until quanteda v1.1, our users needed to have R 3.4.0 installed, but we relax the requirement to R 3.1.0, because people working in companies or other large organizations often do not have latest version of R in their computers, and therefore cannot use our package. To investigate why quanteda requires R 3.4.0 quickly, I wrote […]
PhD thesis is now archived
My PhD thesis titled Measuring bias in international news: a large-scale analysis of news agency coverage of the Ukraine crisis has been archived electronically in the LSE Library and become publicly available. This thesis is a compilation of research papers, three of which have already been published, but its grand conclusion is more than a […]
Quanteda Tutorials
We launched the Quanteda Tutorials website for a workshop Introduction to Quantitative Text Analysis using Quanteda held at the WZB Berlin Social Science Center on 31st January. The website is still work-in-progress, but it already covers all the important Quanteda functions.
Release of Quanteda version 1.0
We have announced the release of quanteda version 1.0 at the London R meeting on Tuesday. I thank all the organizers and 150+ participants. In the talk, I presented the performance comparison with R and Python packages, but I actually compared the performance with its earlier CRAN versions to show how the package evolved to […]
A new paper on Russian media’s coverage of protests in Ukraine
A paper ‘Russian Spring’ or ‘Spring Betrayal’? The Media as a Mirror of Putin’s Evolving Strategy in Ukraine that I co-authored with Tomila Lankina as part of the British Academy-funded project appeared in Europe-Asia Studies. We analyse Russian state media’s framing of the Euromaidan protests using a novel Russian-language electronic content-analysis dictionary and method that […]
Historical analysis of NYT using web API
We usually use commercial database such as Nexis to download news stories in the past, but you should use New York Times APIs if you want to do historical analysis of news content. We can search NYT news articles until 1851 through the API, and it is free for anyone! We can only download meta-data, […]
What is the best SVD engine for LSA in R?
I use latent semantic analysis (LSA) to extract synonyms from a large corpus of news articles. I was very happy with Gensim‘s LSA function, but I was not sure how to do LSA in R as good as in Python. There is an R package called lsa, but it is unsuitable for large matrices, because […]
Workshop on text analysis at ESSCE in London
I and Ken Benoit will convene a half-day workshop at European Symposium Series on Societal Challenges in Computational Social Science in November. We will explain how quanteda can be allied to different types of textual data in dictionary-based analysis. The material for the workshop contains the most updated examples.
Applying LIWC dictionary to a large dataset
LIWC is a popular text analysis package developed and maintained by Pennebaker et al. The latest version of the LIWC dictionary was released in 2015. This dictionary seems more appropriate than classic dictionaries such as the General Inquire dictionaries for analysis of contemporary materials, because our vocabulary changes over years. However, LIWC did not work […]