Fractal Language Modelling by Universal Sequence Maps (USM)
Summary AI-generated
- TL;DR
- Researchers have improved a fractal-based method called Universal Sequence Maps to convert symbolic sequences, like DNA or text, into precise numerical representations without losing contextual information.
- Problem
- Modern language models need efficient ways to convert sequences of symbols into numbers while preserving the exact order and context of those symbols. Existing encoding methods often struggle to uniquely map these sequences at multiple scales without losing crucial contextual details.
- Method
- The authors use Universal Sequence Maps (USM), a technique based on chaos theory that maps sequences into a geometric, fractal space using forward and backward iterations. This approach allows researchers to calculate sequence similarities and frequency patterns mathematically without having to repeatedly recalculate the coordinates.
- Results
- The study resolves initial seeding biases in the mapping process, ensuring that every unique sequence aligns perfectly with a specific numerical position. Additionally, the researchers discovered that this method naturally stabilizes into a steady-state sequence representation, demonstrating its effectiveness on genomic data and its readiness for larger alphabets.
- Takeaways
- First, resolving seeding biases makes fractal sequence encoding mathematically precise and reliable. Second, the method allows for flexible sequence analysis, such as calculating fractional pattern lengths. Finally, while demonstrated on DNA, the technique easily scales to any alphabet size, including natural language.
- For industry
- For businesses working with sequential data, this research offers a highly efficient, mathematically precise way to encode data like text, genetic sequences, or transaction logs. By representing sequences as stable numerical coordinates, companies can perform fast similarity searches and pattern analyses without the heavy computational overhead of traditional language models.
- Why it matters
- This work advances the foundational tools used in bioinformatics and natural language processing, paving the way for more accurate genomic analysis and efficient AI models. By providing a mathematically rigorous alternative to standard embeddings, it supports sustainable AI development through computationally lighter sequence representation techniques.
Abstract
Motivation: With the advent of Language Models using Transformers, popularized by ChatGPT, there is a renewed interest in exploring encoding procedures that numerically represent symbolic sequences at multiple scales and embedding dimensions. The challenge that encoding addresses is the need for mechanisms that uniquely retain contextual information about the succession of individual symbols, which can then be modeled by nonlinear formulations such as neural networks. Context: Universal Sequence Maps(USM) are iterated functions that bijectively encode symbolic sequences onto embedded numerical spaces. USM is composed of two Chaos Game Representations (CGR), iterated forwardly and backwardly, that can be projected into the frequency domain (FCGR). The corresponding USM coordinates can be used to compute a Chebyshev distance metric as well as k-mer frequencies, without having to recompute the embedded numeric coordinates, and, paradoxically, allowing for non-integers values of k. Results: This report advances the bijective fractal encoding by Universal Sequence Maps (USM) by resolving seeding biases affecting the iterated process. The resolution had two results, the first expected, the second an intriguing outcome: 1) full reconciliation of numeric positioning with sequence identity; and 2) uncovering the nature of USM as an efficient numeric process converging towards a steady state sequence embedding solution. We illustrate these results for genomic sequences because of the convenience of a planar representation defined by an alphabet with only 4 tokens (the 4 nucleotides). Nevertheless, the application to alphabet of arbitrary cardinality was found to be straightforward.