initial vault

main/общее заметки/100 ml вопросов.md

@@ -0,0 +1,11 @@
+#ml #учеба #собес
+https://dynamic-epoch-4bb.notion.site/100-questions-about-NLP-549ccde0d81a4689b5635888b9d0d7e6
+
+## [[TF-IDF]]
+1. Напишите TF-IDF с нуля? [[ответ 1]]
+2. Что такое нормализация в TF-IDF? [[ответ 2]]
+3. Зачем вы вообще знаете про TF-IDF в наше время и как можете использовать в сложных моделях? [[ответ 3]]
+
+
+
+2026-07-19

main/общее заметки/Gibbs sampling.md

@@ -0,0 +1,7 @@
+#учеба #math 
+[[Method Monte-Carlo]] [[Marcov chain]]
+Gibbs Sampling is a potent Markov Chain Monte Carlo (MCMC)
+
+
+Sources: 
+https://www.geeksforgeeks.org/machine-learning/what-is-gibbs-sampling/

main/общее заметки/IELTS.md

@@ -0,0 +1,2 @@
+#учеба #англ
+[сайт с мок тестами](https://ieltsonlinetests.com/ielts-exam-library?skill=listening)

main/общее заметки/Lemmatization.md

@@ -0,0 +1,7 @@
+#ml #учеба Related to [[data mining]], [[Stemming]]
+
+ Lemmatization - way of group morphologically related word
+Result of lemmatization must be found in dictionary 
+WordNetLemmatizer in nltk
+
+Source: https://www.ibm.com/think/topics/stemming-lemmatization?regionCode=us&languageCode=en&cm-history=us-en

main/общее заметки/New words dict.md

@@ -0,0 +1,6 @@
+#учеба #англ 
+albeit - "even though" or "although" (но все же)
+factorization  - mathematical process of breaking down a complex entity (like a number or algebraic expression) into a product of simpler entities (factors) that multiply together to yield the original (разложение на множители/факторы)
+consonant - согласные (vowel - гласные)
+erroneously - ошибочно (удивлен, что такое слово вообще есть)
+eigenvectors - собственные вектора

main/общее заметки/SVD (singular value decomposition).md

@@ -0,0 +1 @@
+#ml #учеба #math 

main/общее заметки/Stemming.md

@@ -0,0 +1,13 @@
+#ml #учеба  [[data mining]]
+![[Pasted image 20260719174012.png]]
+
+Main function to reduce word to it's original (root) form - stem (lemma).
+### Lovins stemmer 
+Rules only, remove suffixes, main problem is over removal of suffixes the, those, them - the, love - lov and so on.
+### Porter stemmer
+Classify characters as vowels and consonant. Group sequential v and c and remove them by some rules (have rules, so still have some errors)  Implementation in nltk. 
+### Snowball stemmer 
+Updated porter with new rules, stop word and including other languages (including Russian) also have stop words list, such as  _the_, _a_, _being_, and the like. Also has implementation in nltk.
+
+
+Source: https://www.ibm.com/think/topics/stemming?regionCode=us&languageCode=en&cm-history=us-en

main/общее заметки/TF-IDF.md

@@ -0,0 +1,17 @@
+TF - term frequency. IDF - inverse document frequency 
+![[Pasted image 20260719155922.png]]
+
+![[Pasted image 20260719155956.png]]
+
+```python
+from sklearn.feature_extraction.text import TfidfVectorizer
+```
+Usage
+- keywords extraction
+- text clustering (kmeans on tf-idf matrix)
+
+Initially - modification of bag of word, used to filter out useless (less important for meaning words) 
+
+Sources:
+https://habr.com/ru/companies/otus/articles/755772/
+https://www.geeksforgeeks.org/machine-learning/understanding-tf-idf-term-frequency-inverse-document-frequency/

main/общее заметки/Topic modeling.md

@@ -0,0 +1,17 @@
+#ml #учеба
+
+Topic modeling - unsupervised, provides set of topics of docs. In some way modification of [[TF-IDF]] to deal with synonyms and polysemy.
+Important to use [[Stemming]] and [[Lemmatization]] to reduce number of unique words in dictionary of documents. 
+
+LSA (latent semantic analysis or latent semantic indexing) 
+Has to more matrixes Term x Term - number of documents, there terms co-ocur. Document x Document - number of terms each doc has in common 
+Document x term - number of terms in each doc 
+Use [[SVD (singular value decomposition)]] on term x doc Reduce dimensions and use cosine similarity of this resulting matrixes to measure resemblance of documents. 
+
+LDA (Latent Dirichlet allocation)
+probabilistic algorithm. Using doc x term matrixes (freq and co-ocurence) - generate table of probabilities for each keyword in each doc (topic distribution).
+While assigning topic to words it uses [[Gibbs sampling]]
+
+sources:
+https://maartengr.github.io/BERTopic/getting_started/ctfidf/ctfidf.html - bertopic lib
+https://huggingface.co/blog/bobxwu/fastopic 

main/общее заметки/МФТИ учебники.md

@@ -0,0 +1,3 @@
+Вышмат 
+https://mipt.ru/institute-departments/kafedra-vysshey-matematiki/study_docs/books_lections - Иванов и тд
+https://lib.mipt.ru/book/l/000248050008100a2db5dfecfa332c54/Beklemishev-DV-Kurs-analiticheskoi-geometrii-i-lineinoi-algebry.pdf - бек

main/общее заметки/ответы на 100/ответ 2.md

@@ -0,0 +1 @@
+Нормализация нужна,  чтобы мы не переоценивали tf в длинных документах. В итоге нормализация - деление tf на длину документа (кол-во term)

main/общее заметки/ответы на 100/ответ 3.md

@@ -0,0 +1,3 @@
+Зачем вы вообще знаете про TF-IDF в наше время и как можете использовать в сложных моделях?
+TF-IDF - быстрый способ векторизации текста, который можно использовать для проверки гипотез
+Выделение фичей (так как можем получать важность слова и можем добавлять его)