Depot student Dominik has already provided a good overview of how to export data from the ING depot via the ExtraETF workaround. However, not every tool can handle the CSV export properly. For example, DivvyDiary immediately recognized the relevant columns, but the balances didn’t match. The reason for this is that CSV files can vary significantly, as can the data within them. Sometimes, columns aren’t separated by a comma but by a semicolon. And while the difference between 1,000.00 and 1.000,00 might seem minor to us, for DivvyDiary, a 1000 turned into a 1 because the thousands separator was treated as a decimal point.
The solution: As much as I dislike working with Excel, if you open the CSV file in Excel and then save it again as a CSV, even DivvyDiary (and many other tools) can handle it.
The problem with the code above lies in the use of the pipe operator (|>), right before ggplot. ggplot2 is not natively supported with the R-specific pipe (|>), as used here. However, ggplot2 works seamlessly with the Magrittr pipe (%>%) from the dplyr package. Here is the correct usage:
library(ggplot2) library(dplyr)
mtcars %>% ggplot(aes(x = mpg, y = hp)) + geom_point()
Alternatively, the data must be explicitly passed to ggplot, as shown here:
It was a bit surprising. I had recently emailed with Jim Sterne when it came to the German branch. The DAA had also contributed a foreword to my web analytics book. It’s a bit of a shame.
For those who don’t know: The DAA was previously the WAA, the Web Analytics Association, and it created the most widely used definition of web analytics. Although that definition has long been missing from the website, most researchers who copy quotes from other papers didn’t seem to care.
But how is it possible that such an organization, despite the importance of data, is shutting down? It could be, for example, because many have installed Google Analytics & Co., but the data is not actually being used. In my last paper, which unfortunately isn’t public yet, it was found that most users don’t even realize that embedding the GA code alone isn’t enough to work data-driven. And maybe it’s also a bit due to the DAA itself, that it didn’t manage to make its relevance clear.
I had only been a member out of nostalgia in recent years. I had used my student status to lower the membership fees a bit.
Three years ago, for the 15th anniversary of this blog, I moved from WordPress to Hugo. Super-fast pages, everything in R, actually a cool thing. But in reality, it wasn’t that great. I always needed an R environment, which I didn’t always have. Git drove me crazy at times. And some problems were just impossible to troubleshoot. So now I’ve moved back again. Maybe the rankings will return as well, which I lost after the move.
The terms Artificial Intelligence (AI), Machine Learning, Data Science, Data Mining, Statistics, and Large Language Models (LLMs) are often used interchangeably or misunderstood. Clearly differentiating between these concepts helps you navigate discussions and make informed decisions in data-driven contexts.
Artificial Intelligence (AI)
AI encompasses techniques and algorithms that enable computers to perform tasks traditionally requiring human intelligence, such as reasoning, decision-making, and pattern recognition.
Machine Learning (ML)
ML is a subset of AI where systems learn from data to improve decision-making or predictions without explicit programming. Applications include recommendation engines, fraud detection, and image recognition.
Data Science
Data Science is an interdisciplinary field combining scientific methods, processes, and systems to extract actionable insights from data. It integrates domain expertise, statistical techniques, and data analysis skills to make informed business decisions.
Data Mining
Data Mining involves exploring large datasets to discover meaningful patterns, correlations, or trends. Common applications include customer segmentation, market basket analysis, and anomaly detection.
Statistics
Statistics forms the mathematical basis for Data Science and Machine Learning. It includes methods for collecting, analyzing, interpreting, and presenting data, ensuring rigorous analysis and reliable results.
Large Language Models (LLMs)
Large Language Models are a specialized, advanced type of Machine Learning model that process and generate natural language text. They excel at tasks such as content summarization, text generation, language translation, and interactive dialogue (e.g., ChatGPT).
The Connection Between These Terms:
Artificial Intelligence is the overarching goal of creating systems that simulate human intelligence.
Machine Learning is a key approach to achieving AI through data-driven learning.
Data Science covers the broader methodology of turning data into actionable insights.
Data Mining focuses specifically on finding meaningful patterns in large datasets.
Statistics underpins these fields, providing the mathematical rigor needed for trustworthy analysis.
Large Language Models are an advanced application of Machine Learning, focusing on language understanding and generation.
Why clarity matters
While “Data Science” has dominated conversations in recent years, many discussions have now shifted towards AI and especially Large Language Models. However, even with the buzz around AI, it’s important to remember that successful projects often rely heavily on foundational Data Science and robust statistical methods. Clearly distinguishing these concepts allows you to harness the full potential of data-driven solutions and avoid common misconceptions.
In fact, the first project has now been partially collected, with €262.72 of the €500 returned.
Otherwise, not much has changed since my last update. However, I decided to invest again. I withdrew 2/5 of my investment amount, and while the remaining amount is still significant, it no longer makes up too large a portion of my portfolio. I also put a stop to the reinvestment, at least as best as I could. Unfortunately, at Estateguru, you can’t specify that you want to invest only a certain amount, so that the interest is always available. Instead, you can only specify that you want to keep a certain amount invested, which isn’t ideal. For example, if I had invested €10,000 and wanted to reserve €100 each month, but then received a payment of €300, I would quickly end up investing more than €10,000, even though that’s not what I wanted. The support wasn’t particularly helpful in this case.
Estateguru does offer the option to download your portfolio data, which allows me to take a closer look at what actually went wrong:
data %>% group_by(Country, Status) %>% ggplot(., aes(fill = Status, x = as.factor(Country))) + geom_bar() + theme_minimal() + xlab("Land")
As you can see here, I had fewer projects in Germany than in Estonia, for example, but most of the projects in Germany have defaulted. That’s quite alarming. It looks even worse when you look at the actual amounts.
data %>% group_by(Country, Status) %>% ggplot(., aes(fill = Status, y = `Initial Principal`, x = as.factor(Country))) + geom_bar(stat = "identity") + theme_minimal() + xlab("Land") + ylab("Darlehensbetrag in Euro")
Is there a correlation between the interest rate and the “defaulted” status, meaning did I take on riskier loans in my “greed” that were characterized by higher interest rates? Let’s first visualize several variables:
We definitely see the one outlier where I invested €2,500 at around 11%. It also seems that defaults are primarily associated with higher interest rates, except for the German projects, where I have defaults across the board. However, it doesn’t quite fit, because in some projects, you could invest over multiple stages:
Apparently, I was particularly bold in Germany, thinking that loans there were safer, and as a result, I repeatedly exceeded the limit of €500 per project that I had set for myself. Calculating a statistically significant difference would be the task. But let’s start in a different way first:
data %>%
filter(Status == "Repaid" | Status == "In Default") %>%
group_by(Status) %>%
summarize(mean_interest = mean(`Interest Rate`), median_interest = median(`Interest Rate`))
## # A tibble: 2 × 3
## Status mean_interest median_interest
## <chr> <dbl> <dbl>
## 1 In Default 10.7 10.5
## 2 Repaid 10.0 10
The Shapiro-Wilk test helps us check the normality of the data.
repaid <- filter(data, Status == "Repaid") In fact, the first project has now been partially collected, with €262.72 of the €500 returned.
Estateguru: Project partially recovered. Otherwise, not much has changed since my last update. However, I’ve decided to invest again. I withdrew 2/5 of my investment amount, and while the remaining amount is still substantial, it no longer makes up too large a portion of my portfolio. Additionally, I’ve put a stop to the reinvestment, at least as best as I could. Unfortunately, with Estateguru, you can’t specify that you only want to invest a certain amount so that the interest is always available. Instead, you can specify that you want to keep a certain amount invested, which isn’t ideal. For example, if I had invested €10,000 and wanted to reserve €100 every month, but then received a payment of €300, I would quickly end up investing more than €10,000, even though I didn’t want that. The support wasn’t very helpful in this regard.
Estateguru offers the option to download your portfolio data, which allows me to take a closer look at what actually went wrong:
data %>% group_by(Country, Status) %>% ggplot(., aes(fill = Status, x = as.factor(Country))) + geom_bar() + theme_minimal() + xlab("Land")
As you can clearly see here, I had fewer projects in Germany than in Estonia, for example, but most of the projects in Germany have defaulted. That's quite alarming. It looks even worse when you look at the actual amounts.
data %>% group_by(Country, Status) %>% ggplot(., aes(fill = Status, y = `Initial Principal`, x = as.factor(Country))) + geom_bar(stat = "identity") + theme_minimal() + xlab("Land") + ylab("Darlehensbetrag in Euro")
Is there a correlation between the interest rate and the "defaulted" status, meaning did I take on riskier loans in my "greed" that were characterized by higher interest rates? Let's first visualize several variables:
We definitely see the one outlier where I invested €2,500 at around 11%. It also appears that defaults are primarily associated with higher interest rates, except for the German projects, where I have defaults across the board. However, this doesn't quite fit, because in some projects, you could invest over multiple stages:
Apparently, I was particularly bold in Germany, thinking that loans there were safer, and as a result, I repeatedly exceeded the €500 limit I had set for myself per project. Calculating a statistically significant difference would be the task. But let's start in a different way first:
data %>% filter(Status == "Repaid" | Status == "In Default") %>% group_by(Status) %>% summarize(mean_interest = mean(`Interest Rate`), median_interest = median(`Interest Rate`))
## # A tibble: 2 × 3 ## Status mean_interest median_interest ## <chr> <dbl> <dbl> ## 1 In Default 10.7 10.5 ## 2 Repaid 10.0 10
The Shapiro-Wilk test helps us check the normality of the data.
Interest Rate`in_default <- filter(data, Status == "In Default")
In fact, the first project has now been partially recovered, with €262.72 of the €500 returned.
Estateguru: Project partially recovered.
Otherwise, not much has changed since my last update. However, I’ve decided to invest again. I withdrew 2/5 of my investment, and while the remaining amount is still significant, it no longer makes up too large a portion of my portfolio. Additionally, I’ve put a stop to the reinvestment, at least as best as I could. Unfortunately, with Estateguru, you can’t specify that you only want to invest a certain amount, so that the interest is always available. Instead, you can only set a certain amount to be always invested, which isn’t ideal. For example, if I had invested €10,000 and wanted to reserve €100 every month, but then received a payment of €300, I would quickly end up investing more than €10,000, even though that’s not what I wanted. The support wasn’t very helpful in this case.
Estateguru offers the option to download your portfolio data, which allows me to take a closer look at what actually went wrong:
data %>% group_by(Country, Status) %>% ggplot(., aes(fill = Status, x = as.factor(Country))) + geom_bar() + theme_minimal() + xlab("Land")
As you can clearly see here, I had fewer projects in Germany than in Estonia, for example, but most of the projects in Germany have defaulted. That’s quite alarming. It looks even worse when you look at the actual amounts.
data %>% group_by(Country, Status) %>% ggplot(., aes(fill = Status, y = `Initial Principal`, x = as.factor(Country))) + geom_bar(stat = "identity") + theme_minimal() + xlab("Land") + ylab("Darlehensbetrag in Euro")
Is there a correlation between the interest rate and the "defaulted" status, meaning did I take on riskier loans in my "greed" that were characterized by higher interest rates? Let's first visualize several variables:
We definitely see the one outlier where I invested €2,500 at around 11%. It also seems that defaults are primarily associated with higher interest rates, except for the German projects, where I have defaults across the board. However, this doesn’t quite fit, because in some projects, you could invest over multiple stages:
Apparently, I was particularly bold in Germany, thinking that loans there were safer, and as a result, I repeatedly exceeded the €500 limit I had set for myself per project. Calculating a statistically significant difference would be the task. But let's start in a different way first:
data %>% filter(Status == "Repaid" | Status == "In Default") %>% group_by(Status) %>% summarize(mean_interest = mean(`Interest Rate`), median_interest = median(`Interest Rate`))
## # A tibble: 2 × 3 ## Status mean_interest median_interest ## <chr> <dbl> <dbl> ## 1 In Default 10.7 10.5 ## 2 Repaid 10.0 10
The Shapiro-Wilk test helps us check the normality of the data.
cat("P-value for Repaid group:", shapiro_test_repaid$p.value, "\n")
## P-value for Repaid group: 1.143358e-08
cat("P-value for In Default group:", shapiro_test_in_default$p.value, "\n")
## P-value for In Default group: 6.078673e-05
The p-values are significant (below 0.05), indicating that the data is not normally distributed. Therefore, the Mann-Whitney U test is used, a non-parametric test, to compare the interest rates of the two groups.
wilcox_test <- wilcox.test(repaid, in_default, alternative = "two.sided") cat("P-value for Mann-Whitney U test:", wilcox_test$p.value, "\n")
## P-value for Mann-Whitney U test: 6.66547e-08
The p-value is significant, meaning it's below 0.05, indicating that there is a significant difference in interest rates between repaid and defaulted loans. This analysis was done across the entire portfolio. Now, how does this look by country?
countries <- unique(data$Country)
# Function to analyze each country analyze_country <- function(country) { cat("Analyse für", country, ":\n")
# Filter data by country and status data_df <- data %>% filter(Country == country) %>% filter(Status %in% c("Repaid", "In Default"))
# Check if there is enough data for both categories if (nrow(data_df) > 0 & length(unique(data_df$Status)) > 1) {
cat("Mann-Whitney U Test-Ergebnis: W =", test$statistic, ", p-value =", test$p.value, "\n\n") } else { cat("Nicht genug Daten für die Analyse.\n\n") } }
# Analyze each country for (country in countries) { analyze_country(country) }
## Analyse für Estonia : ## Mann-Whitney U Test-Ergebnis: W = 77 , p-value = 0.02871484 ## ## Analyse für Germany : ## Mann-Whitney U Test-Ergebnis: W = 101 , p-value = 0.5058534 ## ## Analyse für Lithuania : ## Mann-Whitney U Test-Ergebnis: W = 224.5 , p-value = 3.126943e-06 ## ## Analyse für Finland : ## Mann-Whitney U Test-Ergebnis: W = 54 , p-value = 0.8649381 ## ## Analyse für Spain : ## Nicht genug Daten für die Analyse. ## ## Analyse für Portugal : ## Nicht genug Daten für die Analyse. ## ## Analyse für Latvia : ## Mann-Whitney U Test-Ergebnis: W = 12 , p-value = 0.04383209
In fact, the difference in Germany is not significant. So, it turns out I wasn't as greedy as I thought after all 🙂
Now, what if I had only invested €50 in each loan instead of sometimes investing much more? How would I be doing today?
It's very clear here that my strategy of spending more on certain projects didn't work out well. It would have been better to invest more evenly and diversify my investments. That's exactly what I'm doing differently now.
There are a few topics that have been on my mind for decades, one of which is how to store and retrieve information and knowledge in a meaningful way—essentially, a Memex. In the 90s, I was a fan of Apple’s HyperCard, and since then, I’ve tried many things, but nothing has proven reliable. Data formats have disappeared (like Apple’s HyperCard), and even though the technical possibilities today are much broader than back then, I don’t have the impression that they have solved the fundamental problem. There is no quick fix that reads articles and books for you and also makes you understand them. I don’t trust summary services like Blinkist, not to mention that they don’t help me with scientific articles either.
In recent months, I’ve turned again to a system that I had used in the 90s but abandoned in favor of HyperCard: the Zettelkasten. Not just any box with index cards, but a Zettelkasten following the Luhmannian principle. Niklas Luhmann was one of the most significant sociologists of the 20th century, with an incredible number of academic publications to his name. His Zettelkasten, which actually consists of several boxes, contains about 90,000 slips and is being digitized at Bielefeld University. Luhmann attributed his productivity to this Zettelkasten system, and after several months using it, I can understand why. So how does the Zettelkasten work?
First of all, there are a few fundamental principles:
The quality of a paper depends on what has already been written. In an ideal world, the thoughts in the Zettelkasten are already formulated well enough to be directly incorporated. The goal of the Zettelkasten, therefore, is to generate insights that are worth publishing.
Everything you read is processed and placed into the Zettelkasten. Rather than following a strict plan (as is often recommended when writing academic papers), everything goes into the Zettelkasten and is, if possible, linked together right away. Luhmann himself always wrote his slips with the thought in mind of how they would fit into the existing notes. But new ideas can also emerge from the Zettelkasten at any time.
Writing about everything instead of starting directly with a hypothesis also has the advantage of reducing exposure to confirmation bias, which causes one to ignore anything that might challenge the hypothesis.
Highlighting in a text is useless, the learning effect is zero. According to Luhmann, thinking cannot happen without writing. Handwritten notes are preferable to those typed on a computer, as they tend to capture the essence better, thus facilitating understanding.
The Zettelkasten is not an archive, nor is it an idea graveyard like Moleskine notebooks.
There is no information hierarchy; thoughts can simply be inserted into Luhmann’s system wherever they fit. For example, if you have a note numbered “1,2,1” and another “1,2,2,” you can just insert a “1,2,1,a” if a thought is missing here. There’s no need to pre-think endlessly about what the best structure is.
Self-discipline is more important than IQ. A smart working environment ensures that you don’t face resistance from the start.
Daily work with the Zettelkasten leads to a new KPI for knowledge workers: How many notes are created each day!
How exactly does the Zettelkasten work?
When reading a text, “Literature Notes” are created, which contain only your own thoughts about the text, written in your own words.
These then lead to “Permanent Notes,” which are added to the Zettelkasten. The Literature Notes are discarded (this is also why I sent back my Scribe, since the Literature Notes could not be created with it).
Additionally, there are “Fleeting Notes,” which contain all the ideas you have.
The Zettelkasten system is highly minimalist, with no fancy notebooks or software tools. The reduction, or even restriction, to what is essential also stimulates creativity and thinking (see Stokes 2001 and Rheinberger 1997). Of course, there are software solutions like The Archive, and they would have advantages for me: I don’t always have my Zettelkasten with me. On the other hand, I’ve learned that when I’m sitting in front of a computer or iPad, I tend to get distracted. Therefore, I now carry index cards with me.
Here is an interesting video about Luhmann’s Zettelkasten, where he explains it himself starting at minute 37:26:
And here is the researcher who is now delving into the Zettelkasten:
Apple is no longer a computer manufacturer, Google is no longer a search engine, and Amazon is no longer just an online shopping mall. What these companies have in common is that they conquer other areas of life from their respective starting points. The effects are already foreseeable.
In addition to the search engine, Google offers e-mail, a mobile phone operating system and mobile phones, a voice-operated assistant, a computer operating system, a browser, self-driving cars, a photo service and other services in the cloud, music, movies, home surveillance systems (Nest), YouTube and much more. Apple offers mobile phones, computers, iPads, software, a music service, a SmartWatch, headphones, a voice-operated assistant, a TV service and a cloud solution, and much more. Among other products, Amazon offers food delivery, its own products, a voice-operated assistant, video conferencing hardware, a music service, a film service and is also the largest cloud provider. And much more.
At first glance, the services may each be a colorful potpourri, but if you take a closer look, the three big digital heavyweights (Facebook excluded from the GAFA acronym) are tinkering with life operating systems. Our lives will increasingly be supported by digital systems, and Apple, Amazon and Google are working to make as much of these systems available as possible. A system that links digital life coherently and coherently could be called a life operating system.
This is especially evident when you want to switch from one system to another or have to exchange data with another system. This is already difficult in some cases. Android phone and MacBook? It’s no fun. Switching from an Android to an Apple phone? No problem at first, there is an assistant. But the calendar? And the contacts? How do I get the photos into Apple’s elegant Photos app? It is only really efficient when all digital activities are designed with Apple products. Photos sync across multiple devices, as do files, no matter how much storage space you have on each device.
At Google, the assistant learns. The more Google products are used, the better suggestions are generated. Traffic jam on the way to work? No problem, the Android phone warns its user based on historical data and the current traffic situation. Here, too, synchronization is carried out across several devices.
Amazon still seems to be lagging behind. But the strategy of selling hardware at a discount, as in the case of tablets, helps to bind customers to the Amazon universe.
And so a new class society could soon arise, which could result from the use of the respective life operating system:
Those who can afford it use the Apple system. It’s expensive, but time-saving and chic.
The Google system will be somewhat cheaper, even if Google tries its hand at the upper price segments with the Pixel devices. It is the system for the masses.
The cheapest will be an Amazon system. It doesn’t offer the fastest hardware, but everything you need is included.
Those who want to remain independent work with Linux computers and free software. The rebels. They pay primarily with lifetime, but also retain control over their data.
However, it will only get really exciting when the AI-based assistants are really good. If they can then use all the information they collect about a user via several different services, then real support like in “Her” would only be possible. And then it will be even harder to switch from one system to the next.
But perhaps the Internet giants will open up and provide each other with interfaces. Siri could learn from Gmail and Alexa from the YouTube playlist. But that doesn’t sound very likely. Instead, it looks more like we will be trapped in the respective systems because switching would bring too many disadvantages.
This also has very practical implications: What if an Apple woman wants to move in with an Amazon guy? It cannot control its hardware and vice versa. Or will there be no more dating between classes?
