Let’s revise Natural Language Processing — 1

AAWEG I *We put ghosts in machines *

13 min readMay 11

--

(1) What is a token? What is tokenisation? And n-gram tokenisation.

In natural language processing (NLP), a token refers to a unit of text that is treated as a single meaningful entity. It can be a word, a sentence, or even smaller components, such as characters or subwords. Tokenization is the process of splitting a text into these individual tokens.

AAWEG I *We put ghosts in machines *

Written by AAWEG I We put ghosts in machines

linkedin.com/in/aaweg-i | github.com/the-ogre

More from AAWEG I We put ghosts in machines

AAWEG I *We put ghosts in machines *

Time Series Forecasting: A Comparative Analysis of SARIMAX and XGBoost Algorithms

SARIMAX (Seasonal Autoregressive Integrated Moving Average with Exogenous Variables) and XGBoost are both popular algorithms used for time…

3 min readMay 14

--

AAWEG I *We put ghosts in machines *
in
Aaweg Interview

Can you explain Jaccard’s Index? How does it differ from Dice Coefficient?

In object detection, there are two distinct tasks to measure:

6 min readNov 27, 2022

--

AAWEG I *We put ghosts in machines *
in
Aaweg Interview

Can you tell us something about ‘Global Average Pooling’?

A typical classifier, except the last two/few layers, is nothing but a feature extractor.

3 min readNov 29, 2022

--

AAWEG I *We put ghosts in machines *
in
Literary Impulse

Exploring the Fragility of Power and Empire: A Critical Reading of Ryan Keating’s ‘Crushing Snails’…

Delve into the thought-provoking poem “Crushing Snails” by Ryan Keating, which vividly describes the interconnectedness of ancient empires…

3 min read6 days ago

--

See all from AAWEG I *We put ghosts in machines *

Recommended from Medium

Two stochastic parrots sitting on a chain of large language models: LangChain

Leonie Monigatti
in
Towards Data Science

Getting Started with LangChain: A Beginner’s Guide to Building LLM-Powered Applications

A LangChain tutorial to build anything with large language models in Python

12 min readApr 25

--

Dr. Roi Yehoshua
in
Towards Data Science

Mastering Logistic Regression

From theory to implementation in Python

17 min readMay 20

--

Lists

What is ChatGPT?

9 stories57 saves

Staff Picks

323 stories82 saves

Matt Crooks
in
Towards Data Science

A Review of Propensity Score Modelling Approaches

A review of different approaches to using propensity scores in causal inference modelling

11 min readMay 17

--

Naveen Mathews Renji

Word Embeddings: Unveiling Semantic Relationships in Natural Language Processing

In the realm of Natural Language Processing (NLP), the way we represent words is of paramount importance. Our words are our raw data, and…

5 min readMay 14

--

Gabe Araujo, M.Sc.
in
Level Up Coding

🐼Introducing PandasAI: The Generative AI Python Library 🐼

Pandas AI is an additional Python library that enhances Pandas, the widely-used data analysis and manipulation tool, by incorporating…

9 min readMay 16

--

Martin Thissen
in
MLearning.ai

Understanding and Coding the Attention Mechanism — The Magic Behind Transformers

In this article, I’ll give you an introduction to the attention mechanism and show you how to code the attention mechanism yourself.

12 min readDec 6, 2022

--

See more recommendations

Help
Status
Writers
Blog
Careers
Privacy
Terms
About
Text to speech