This paper presents a hybrid framework for intelligent malware detection that integrates the Enhanced HawkFish Optimization Algorithm (EHFOA) with the Light Gradient Boosting Ma-chine (LightGBM). The proposed method addresses the challenges of high-dimensional feature spaces, suboptimal model configurations, and real-time detection efficiency by simultaneously performing feature selection and hyperparameter tuning through a multi-objective optimization strategy. EHFOA incorporates biologically inspired behaviors along with chaotic initialization, entropy-based diversity control, and memetic local search refinement to improve convergence stability and search accuracy. The optimized feature subset and classifier configuration are used to train a lightweight yet highly accurate LightGBM model. The framework was evaluated on three benchmark datasets—EMBER, CIC-MalMem2022, and MalImg—and compared against several state-of-the-art models, including PSO-LGBM, GWO-LGBM, CNN, LSTM, DBN, Random Forest, XGBoost, and SVM. Experimental results show that the proposed method achieved a classifica-tion accuracy of 96.87%, precision of 97.12%, recall of 96.45%, and an F1-score of 96.78%, with a false positive rate of 2.18%. The model achieved a 42% feature reduction, reducing the input space to 58 features, and required only 145 seconds for training and 0.012 seconds for inference per sample. Statistical validation confirmed the significance of the performance improvements (p < 0.01), while ROC and precision–recall curves highlighted the model’s robustness under imbalanced class distributions. The results demonstrate that the EHFOA-Light framework offers an effective, scalable, and computationally efficient solution for advanced malware detection.