Discovery Journey: Auto-Generated Activation Functions Using Genetic Programming

 Date:May 21, 2025

Over the past week(s), I’ve embarked on an intensive experimental journey combining genetic programming (DEAP) with neural network activation function design. The result? A utility that evolves custom activation functions capable of outperforming standard choices like ReLU, Swish, or GELU under certain conditions.

This post serves both as a personal milestone and a timestamped record of this research.

🔍 Project Overview

Using DEAP (Distributed Evolutionary Algorithms in Python), I constructed an evolutionary search space made up of elementary mathematical primitives and safe variants of functions like log, sqrt, and divide.

The system:

  • Evolves TensorFlow-compatible activation expressions

  • Benchmarks them in a real model loop (e.g. CNN-based classifier)

  • Reports performance via loss, runtime, and memory usage

  • Filters out invalid or exploding functions during evolution

🧠 Notable Results

One evolved function recently outperformed standard GELU/Swish on accuracy:


tf.add(
    tf.sqrt(tf.maximum(x, 0.0)),
    tf.add(
        tf.add(
            x,
            tf.multiply(tf.maximum(tf.sqrt(tf.maximum(x, 0.0)), 0.0), tf.math.divide_no_nan(
                x, 0.5 * x * (1 + tf.tanh(tf.sqrt(2 / np.pi) * (x + 0.044715 * tf.pow(x, 3))))
            ))
        ),
        tf.math.log(tf.abs(x) + 1e-6)
    )
)

 A GitHub repository will follow shortly.

If you’re working on model optimization, custom layers, or symbolic search: this approach might save weeks of trial-and-error. Pseudo Code

🙏 Stay Tuned

I’ll share more details, benchmarks, and licensing terms in the GitHub release. Feedback welcome. If you’re interested in collaboration or sponsorship, feel free to reach out.

— contact: machinesmartsor@gmail.com  (Paul KP Fung)                   


This is an exploration of automatic activation function discovery using evolutionary strategies with DEAP and TensorFlow. The method was implemented and refined over several iterations. Insights and code support provided by ChatGPT (OpenAI).      


 Attribution & Legal Notes

🔧 Tools & Assistance:

This project and blog post were developed with the assistance of ChatGPT by OpenAI, particularly for code generation, algorithm structuring, and conceptual clarification. Final implementations and ideas are solely attributed to the author.  

📚 Library Attributions:

This work incorporates or references open-source libraries under their respective licenses:

  • DEAP (Distributed Evolutionary Algorithms in Python) – GNU LGPL v3

  • TensorFlow – Apache License 2.0

  • Matplotlib – PSF License

  • NumPy – BSD License

  • psutil – BSD License

Ensure that your own use of these components complies with the terms of each license.

📖 License for This Work:

This blog post and associated pseudocode are released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

This means you are free to:

  • Share — copy and redistribute the material

  • Adapt — remix, transform, and build upon the material

Under the following terms:

  • Attribution — You must give appropriate credit to the original author and indicate if changes were made.

  • NonCommercial — You may not use the material for commercial purposes without explicit permission.

A commercial-use license may be available via contact or linked GumRoad offerings.

⚠️ Disclaimers:

  • This blog post does not constitute legal or patent advice.

  • The material is shared for research, educational, and experimental purposes only.

  • The author does not guarantee performance or fitness for any particular purpose.

Posted on https://artisticsciencedream.blogspot.com/

Copyright © 2025 by Paul KP Fung

Comments

Post a Comment

Popular posts from this blog

Evolving Activation Functions: A Personal Exploration with Transformers

Image
⚠️ Disclaimer (Please Read First) This blog post presents a personal, exploratory experiment using publicly available tools and datasets (e.g., Tatoeba, DEAP, Hugging Face Transformers). It is not a lab-verified study, and results have not undergone peer review or formal statistical validation. The findings, interpretations, and conclusions shared here are based on limited-scale experiments using Google Colab and consumer-level hardware. They are intended for educational and exploratory purposes only. There is no guarantee on the accuracy, stability, or reproducibility of the experimental results. Any interpretations or applications are entirely at the reader’s discretion. Readers are encouraged to replicate, adapt, or challenge the outcomes in more rigorous or production-grade environments. We often tweak our models by adjusting the data, trying new optimizers, or changing the architecture—but how often d...
Read more

Analyzing Stock Trends with Bollinger Bands, RSI, and MACD

Image
  Introduction Successful trading and investing rely on effective stock market analysis tools to identify trends and potential trade signals. Whether you're a beginner learning technical analysis or an experienced trader optimizing strategies , using key technical indicators can enhance decision-making. In this guide, I'll walk you through a trading algorithm framework that fetches real-time stock data and computes essential indicators like Bollinger Bands, RSI (Relative Strength Index), and MACD (Moving Average Convergence Divergence) . You'll also learn how to visualize stock trends for better insights. This stock analysis script is especially useful for swing traders and active investors looking to identify potential trade signals and refine their trading strategies. What This Trading Algorithm Does This automated trading analysis script simplifies the process of retrieving stock market data and analyzing it with technical indicators . The provided pseudo ...
Read more