A hybrid deep learning and ensemble approach for accurate prediction of hepatocellular carcinoma from sequence variants of hepatitis B virus (HBV) X gene

The current study presents a hybrid framework combining deep learning and ensemble learning models to predict hepatocellular carcinoma (HCC) based on sequence variants of the hepatitis B virus (HBV)-X gene. In contrast to traditional single-model approaches, the proposed framework integrates deep learning architectures with data balancing and ensemble learning strategies to enhance predictive robustness and generalization. The dataset comprises 3,360 distinct DNA sequences categorized into HCC and Non-HCC classes, with a significant class imbalance favoring the Non-HCC group. To address this issue, the Synthetic Minority Over-Sampling Technique (SMOTE) was applied to reduce classification bias and improve minority class learning. Two sequence encoding strategies, One-Hot encoding and K-mer encoding (K = 3 and K = 6), were employed to capture nucleotide composition and sequence dependencies. A range of machine learning models, including Multinomial Naive Bayes (MNB), Support Vector Classifier (SVC), Nu-Support Vector Classifier (NuSVC), Logistic Regression (LR), XGBoost, Random Forest (RF), and AdaBoost, as well as deep learning models such as Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory (BiLSTM), were systematically evaluated. To further enhance predictive performance, a hybrid stacking ensemble framework was developed, integrating a CNN–BiLSTM architecture and Random Forest as base learners, with Logistic Regression serving as the meta-classifier to combine their predictions. All methodological steps, model configurations, and evaluation strategies were explicitly defined to ensure reproducibility and facilitate verification. The proposed hybrid stacking ensemble model demonstrated superior performance compared to individual machine learning and deep learning models. One-Hot encoding consistently outperformed K-mer encoding across most models. The best-performing hybrid stacking ensemble achieved an accuracy of 0.9139 and an F1-score of 0.9126, indicating a strong balance between classification performance and generalization. The application of SMOTE further improved model stability and reduced classification bias. These results highlight that the integration of hybrid deep learning models with ensemble learning and class imbalance correction provides a significant improvement over conventional single-model approaches for HBV-related HCC prediction. The proposed framework has potential applications in clinical diagnostic systems for early and accurate detection of HCC. Future work will focus on external validation using independent datasets and further optimization of encoding strategies and advanced deep learning architectures.