Posts

Hi, this is the final post in my series about RAG systems. Here I will look into fine-tuning the generative part of a RAG system. Here is the notebook. But mostly this is for me a method to understand LLM fine-tuning in a superficial and general way. For the records, I looked into search, ANN Algorithms, fine-tuning embedding models, fine-tuning a reranker and evaluation of RAG systems in my previous posts. In RAG, the retrieval step provides the facts. The fine-tuning is about model behavior to prevent that a base model hallucinate or ignore the provided data. As a rule of thumb, retrieval gives the model its memory and fine-tuning gives it its instructions. ...

Hi, Based on my previous post, I looked into fine-tuning a bi-encoder via self-supervised learning. As a logical next step, I now want to fine-tune a cross-encoder for my specific task. This is not as easy as fine-tuning a bi-encoder, because cross-encoders are not designed to be trained with contrastive learning. Therefor I looked into four different approaches to fine-tune a cross-encoder for my specific task: BCE loss BCE loss with hard negatives InfoNCE loss Margin MSE loss BCE loss The simplest approach is to use binary cross-entropy loss to train the model. Here we have two labels, 1 for a positive pair and 0 for a negative pair and treat the relevance as a binary classification problem Passage Re-Ranking with BERT by Nogueira and Cho 2020. This can also be called a pointwise approach. For every query-document pair we calculate the logit, then apply a sigmoid activation twhich will represent the probability of relevance. ...

Hi, Based on my previous post, I need to write a correction. I was wrong. I previously suggested using embeddings directly from pre-trained models. That turns out to be a bad idea, because the training objectives are fundamentally different. As an alternative, I have now explored training an embedding model using Contrastive Learning. Please look at my notebook for the full code and details. The main reason you should not use raw embeddings from pre-trained models is the anisotropy problem. The paper How Contextual are Contextualized Word Representations? from Ethayarajh et al. 2019 shows that the embeddings are clustered in a narrow cone of the vector space, making them almost useless for differentiation. So, if you calculate the cosine similarity between two completely different sentences, the result will be around 0.80. ...

Hi, I decided to add another post to this year’s recap. After I described my three key learnings from my failed startup endeavor, I wasn’t feeling finished or satisfied. Overall I’m not satisfied with what I learned this year. It feels very shallow without any hard skills behind it. In every year of my life, I learned a lot, independent of the later usefulness, but at least I can say, that I moved forward. This year, I don’t feel like this. As Confucius said: “By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest.” My learnings weren’t helpful at all and they came from experience. To sum it up, the ratio of effort and outcome was very low and I definitely can do better. ...

Hi, This year was an interesting learning experience. I tried to commercialize a product, which in the end was much harder than expected. As a short summary, the first attempt failed, because I couldn’t figure out how to sell the product. The pivot failed, because multiple big corporates released a similar product. Also the goal was to get 1-2 customers to build a MVP before looking for funding and so on. So I never incorporated the idea into a company. Basically, I failed already on the first meter, but this saved a lot of time. So I only invested 7 months. 🥲 ...

Hi, Over the last three months, I’ve been working on Agents and LLM workflows. I had the opportunity to do this as part of a residency program at Merantix. I’m deeply thankful for the opportunity and would highly recommend the program to others. Being surrounded by people with shared goals was incredibly motivating and inspiring. I already miss the discussions and the knowledge exchange. Unlike typical work environments, here the participants were truly aligned, and everyone learned from each other. ...

Hi, The implementation of this article is here. RAGs are complex systems. This is obvious, when you try to evaluate them. There are multiple aspects, which need to be checked. Here, I try to look into different approaches to get a better understanding and problems, when facing RAG systems. RAG system evaluation involves two distinct parts: retrieval and generation part. For retrieval, context relevance and noise robustness are key factors in assessing quality, while for generation part, key factors like answer faithfulness, answer relevance, negative rejection, information integration, and counterfactual robustness are important (Gao et al. 2024). ...

Hi, This post is about the approximate nearest neighbor (ANN) algorithm. The code for this post is here, where I provide an example of using a framework and a python implementation. Most python implementation were written with the help of a LLM. I’m amazed, how helpful they are for learning new things. I see them like a drunken professor, which with the right approach will be a very helpful tool. As a next step in understanding RAGs, I want to have a closer look at approximate nearest neighbor algorithms. Basically, the purpose is to find the closest vector to a query vector in a database. Since I’m also interested into the implementation, I follow mostly this amazing blog post. Vector search is the basic component of vector databases and their main purpose. ANN algorithms are looking for a close match instead of an exact match. This loss of accuracy allows an improvement of efficieny, which allows the search through much bigger datasets, high-dimensional data and real-time apllications. ...

Hi, Some time ago, I did a small project on information retrieval. I think, it\s a good idea to share it with all its shortcomings. Here is the code. Sadly, the LLM part doesn’t work with the quantized model, so I commented it out. The project is a small information retrieval of a FAQ, where I want to map the correct answer to a question. In my example, it’s a 1:1 mapping between question and answer, but it also works with multiple answers. ...

DISCLAIMER: This is wrong. Please ignore this post. Hi, Following my first short post about RAGs, I would like to provide a brief overview about embeddings, which are used to find similiar objects in a vector database. To better understand how various transformer models handle different input data types, I created this notebook. I explore therefor, text, image, audio and video data. I’ve chosen to skip the more traditional text embeddings (TF-IDF, Word2Vec or GloVe), because there are already very good tutorials available. Additionally, I plan to discuss the training of embedding models in a separate blog post. For this post, I use mostly pretrained classification models, where I use the last layer before the prediction head as embedding. ...