KitsuVp/FanConections

FANformer: Fourier Analysis Neural Transformer

FANformer is an advanced language model architecture that combines the power of transformers with efficient computational mechanisms for improved performance. The model incorporates Fourier Analysis components to better capture periodicities in language patterns.

Model Description

FANformer introduces several key architectural innovations:

• Fourier Analysis Neural Processing: Captures periodic patterns in data through trigonometric transformations
• Compressed Linear Layers (CoLA): Reduces parameter count by factorizing matrix operations into low-rank approximations
• Hybrid Normalization: Combined Pre-Norm and QKV-Norm strategies for improved training stability
• HyperConnections: Advanced residual connections with dynamic parameters for better gradient flow
• Optimized Flash Attention: Implements efficient attention mechanisms with adaptive normalization

Key Features

• Efficient training through parameter-efficient design
• Strong performance on text generation tasks
• Balance between computational efficiency and model expressivity
• Higher quality outputs by capturing periodic patterns in language

Usage

You can use this model with the Transformers library:

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("KitsuVp/FanConections")
model = AutoModelForCausalLM.from_pretrained("KitsuVp/FanConections", trust_remote_code=True)
model.eval()

# Generate text
input_text = "The FANformer architecture combines"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

# Generate with parameters
outputs = model.generate(
    input_ids, 
    max_length=100, 
    top_p=0.9,
    temperature=0.7,
    num_return_sequences=1
)

# Decode and print result
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

Model Architecture Details

The FANformer model implements a decoder-only transformer architecture with several novel components:

1. FAN Components: Incorporates Fourier Analysis into linear projections to better model periodic data patterns
2. Low-Rank Matrix Factorization: Uses CoLA_FAN and CoLA_Linear layers to reduce parameter count
3. RoPE Positional Embeddings: Implements rotary positional embeddings for better position awareness
4. Progressive Dropout: Implements scaled dropout that increases with network depth
5. Flash Attention with Unpadding: Optimized attention computation for maximum GPU utilization

Training

The model was trained on a mixture of educational web content from FineWeb and mathematical text from FineMath. Training employed:

• Distributed training with multiple GPUs
• The specialized Muon optimizer with Newton-Schulz orthogonalization
• Progressive learning rate scheduling
• Mixed precision (bfloat16) training
• Strategic gradient checkpointing for memory efficiency

Limitations

• Limited context window (1024 tokens)
• May not perform optimally on highly specialized domain content
• Like all language models, can produce incorrect or misleading information

Citation

If you use this model in your research, please cite:

@misc{fanformer2025,
  author = {[Kitsun]},
  title = {FANformer: Fourier Analysis Neural Transformer},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co./KitsuVp/FanConections}}
}

License

This model is released under the Apache 2.0 License.