Skip to content

Commit

Permalink
Merge pull request #511 from dice-group/add_roces_nces2
Browse files Browse the repository at this point in the history 
Add ROCES and NCES2
  • Loading branch information
@alkidbaci
alkidbaci authored Feb 4, 2025
2 parents 3a1b063 + 6424a1f commit 1f36cd0
Show file tree
Hide file tree
Showing 33 changed files with 1,793 additions and 2,239 deletions.
68 changes: 43 additions & 25 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ $$\forall p \in E^+\ \mathcal{K} \models H(p) \wedge \forall n \in E^-\ \mathcal
To tackle this supervised learning problem, ontolearn offers many symbolic, neuro-symbolic and deep learning based Learning algorithms:
- **Drill** → [Neuro-Symbolic Class Expression Learning](https://www.ijcai.org/proceedings/2023/0403.pdf)
- **EvoLearner** → [EvoLearner: Learning Description Logics with Evolutionary Algorithms](https://dl.acm.org/doi/abs/10.1145/3485447.3511925)
- **NCES2** → (soon) [Neural Class Expression Synthesis in ALCHIQ(D)](https://papers.dice-research.org/2023/ECML_NCES2/NCES2_public.pdf)
- **NCES2** → [Neural Class Expression Synthesis in ALCHIQ(D)](https://papers.dice-research.org/2023/ECML_NCES2/NCES2_public.pdf)
- **ROCES** → [Robust Class Expression Synthesis in Description Logics via Iterative Sampling](https://www.ijcai.org/proceedings/2024/0479.pdf)
- **NCES** → [Neural Class Expression Synthesis](https://link.springer.com/chapter/10.1007/978-3-031-33455-9_13)
- **NERO** → (soon) [Learning Permutation-Invariant Embeddings for Description Logic Concepts](https://link.springer.com/chapter/10.1007/978-3-031-30047-9_9)
- **CLIP** → [Learning Concept Lengths Accelerates Concept Learning in ALC](https://link.springer.com/chapter/10.1007/978-3-031-06981-9_14)
Expand Down Expand Up @@ -249,46 +250,47 @@ print(df[[col for col in df if col.startswith('Test-F1') or col.startswith('RT')

Below, we report the average test F1 score and the average runtimes of learners.

| LP | Test-F1-OCEL | RT-OCEL | Test-F1-CELOE | RT-CELOE | Test-F1-Evo | RT-Evo | Test-F1-DRILL | RT-DRILL | Test-F1-TDL | RT-TDL | Test-F1-NCES | RT-NCES | Test-F1-CLIP | RT-CLIP |
|:------------------:|-------------:|--------:|--------------:|---------:|------------:|-------:|--------------:|---------:|------------:|-------:|-------------:|--------:|-------------:|--------:|
| Aunt | 0.614 | 13.697 | 0.855 | 13.697 | 0.978 | 5.278 | 0.811 | 60.351 | 0.956 | 0.118 | 0.812 | 1.168 | 0.855 | 14.059 |
| Cousin | 0.712 | 10.846 | 0.789 | 10.846 | 0.993 | 3.311 | 0.701 | 60.485 | 0.820 | 0.176 | 0.677 | 1.050 | 0.779 | 9.050 |
| Grandgranddaughter | 1.000 | 0.013 | 1.000 | 0.013 | 1.000 | 0.426 | 0.980 | 17.486 | 1.000 | 0.050 | 1.000 | 0.843 | 1.000 | 0.639 |
| Grandgrandfather | 1.000 | 0.897 | 1.000 | 0.897 | 1.000 | 0.404 | 0.947 | 55.728 | 0.947 | 0.059 | 0.927 | 0.902 | 1.000 | 0.746 |91.66
| Grandgrandmother | 1.000 | 4.173 | 1.000 | 4.173 | 1.000 | 0.442 | 0.893 | 50.329 | 0.947 | 0.060 | 0.927 | 0.908 | 1.000 | 0.817 |
| Grandgrandson | 1.000 | 1.632 | 1.000 | 1.632 | 1.000 | 0.452 | 0.931 | 60.358 | 0.911 | 0.070 | 0.911 | 1.050 | 1.000 | 0.939 |
| Uncle | 0.876 | 16.244 | 0.891 | 16.244 | 0.964 | 4.516 | 0.876 | 60.416 | 0.933 | 0.098 | 0.891 | 1.256 | 0.928 | 17.682 |
| LP | Test-F1-OCEL | RT-OCEL | Test-F1-CELOE | RT-CELOE | Test-F1-Evo | RT-Evo | Test-F1-DRILL | RT-DRILL | Test-F1-TDL | RT-TDL | Test-F1-NCES | RT-NCES | Test-F1-NCES2 | RT-NCES2 | Test-F1-ROCES | RT-ROCES | Test-F1-CLIP | RT-CLIP |
|:------------------:|-------------:|--------:|--------------:|---------:|------------:|-------:|--------------:|---------:|------------:|-------:|---------------:|----------:|----------------:|-----------:|----------------:|-----------:|-------------:|--------:|
| Aunt | 0.614 | 13.697 | 0.855 | 13.697 | 0.978 | 5.278 | 0.811 | 60.351 | 0.956 | 0.118 | 0.805 | 0.632 | 0.812 | 1.136 | 0.812 | 1.119 | 0.855 | 14.059 |
| Cousin | 0.712 | 10.846 | 0.789 | 10.846 | 0.993 | 3.311 | 0.701 | 60.485 | 0.820 | 0.176 | 0.608 | 0.628 | 0.680 | 1.177 | 0.695 | 1.086 | 0.779 | 9.050 |
| Grandgranddaughter | 1.000 | 0.013 | 1.000 | 0.013 | 1.000 | 0.426 | 0.980 | 17.486 | 1.000 | 0.050 | 1.000 | 0.507 | 1.000 | 0.955 | 1.000 | 0.917 | 1.000 | 0.639 |
| Grandgrandfather | 1.000 | 0.897 | 1.000 | 0.897 | 1.000 | 0.404 | 0.947 | 55.728 | 0.947 | 0.059 | 0.927 | 0.505 | 0.947 | 0.944 | 0.927 | 0.924 | 1.000 | 0.746 |
| Grandgrandmother | 1.000 | 4.173 | 1.000 | 4.173 | 1.000 | 0.442 | 0.893 | 50.329 | 0.947 | 0.060 | 0.947 | 0.633 | 0.933 | 1.323 | 0.947 | 1.306 | 1.000 | 0.817 |
| Grandgrandson | 1.000 | 1.632 | 1.000 | 1.632 | 1.000 | 0.452 | 0.931 | 60.358 | 0.911 | 0.070 | 0.909 | 0.598 | 0.931 | 1.171 | 0.931 | 1.146 | 1.000 | 0.939 |
| Uncle | 0.876 | 16.244 | 0.891 | 16.244 | 0.964 | 4.516 | 0.876 | 60.416 | 0.933 | 0.098 | 0.854 | 0.538 | 0.891 | 0.948 | 0.891 | 0.905 | 0.928 | 17.682 |


| LP | Train-F1-OCEL | Train-F1-CELOE | Train-F1-Evo | Train-F1-DRILL | Train-F1-TDL | Train-F1-NCES | Train-F1-CLIP |
|:------------------:|--------------:|---------------:|-------------:|---------------:|-------------:|----------------:|----------------:|
| Aunt | 0.835 | 0.918 | 0.995 | 0.837 | 1.000 | 0.804 | 0.918 |
| Cousin | 0.746 | 0.796 | 1.000 | 0.732 | 1.000 | 0.681 | 0.798 |
| Grandgranddaughter | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Grandgrandfather | 1.000 | 1.000 | 1.000 | 0.968 | 1.000 | 0.973 | 1.000 |
| Grandgrandmother | 1.000 | 1.000 | 1.000 | 0.975 | 1.000 | 0.939 | 1.000 |
| Grandgrandson | 1.000 | 1.000 | 1.000 | 0.962 | 1.000 | 0.927 | 1.000 |
| Uncle | 0.904 | 0.907 | 0.996 | 0.908 | 1.000 | 0.884 | 0.940 |
| LP | Train-F1-OCEL | Train-F1-CELOE | Train-F1-Evo | Train-F1-DRILL | Train-F1-TDL | Train-F1-NCES | Train-F1-NCES2 | Train-F1-ROCES | Train-F1-CLIP |
|:------------------:|--------------:|---------------:|-------------:|---------------:|-------------:|----------------:|-----------------:|-----------------:|----------------:|
| Aunt | 0.835 | 0.918 | 0.995 | 0.837 | 1.000 | 0.759 | 0.804 | 0.804 | 0.918 |
| Cousin | 0.746 | 0.796 | 1.000 | 0.732 | 1.000 | 0.680 | 0.696 | 0.728 | 0.798 |
| Grandgranddaughter | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Grandgrandfather | 1.000 | 1.000 | 1.000 | 0.968 | 1.000 | 0.910 | 0.944 | 0.942 | 1.000 |
| Grandgrandmother | 1.000 | 1.000 | 1.000 | 0.975 | 1.000 | 0.923 | 0.941 | 0.944 | 1.000 |
| Grandgrandson | 1.000 | 1.000 | 1.000 | 0.962 | 1.000 | 0.911 | 0.923 | 0.923 | 1.000 |
| Uncle | 0.904 | 0.907 | 0.996 | 0.908 | 1.000 | 0.823 | 0.886 | 0.884 | 0.940 |


### 10-Fold Cross Validation Mutagenesis Benchmark Results
```shell
python examples/concept_learning_cv_evaluation.py --kb ./KGs/Mutagenesis/mutagenesis.owl --lps ./LPs/Mutagenesis/lps.json --path_of_nces_embeddings ./NCESData/mutagenesis/embeddings/ConEx_entity_embeddings.csv --path_of_clip_embeddings ./CLIPData/mutagenesis/embeddings/ConEx_entity_embeddings.csv --max_runtime 60 --report mutagenesis_results.csv
```

| LP | Train-F1-OCEL | Test-F1-OCEL | RT-OCEL | Train-F1-CELOE | Test-F1-CELOE | RT-CELOE | Train-F1-Evo | Test-F1-Evo | RT-Evo | Train-F1-DRILL | Test-F1-DRILL | RT-DRILL | Train-F1-TDL | Test-F1-TDL | RT-TDL | Train-F1-NCES | Test-F1-NCES | RT-NCES | Train-F1-CLIP | Test-F1-CLIP | RT-CLIP |
|:---------|--------------:|-------------:|--------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|--------------:|-------------:|--------:|--------------:|-------------:|--------:|
| NotKnown | 0.916 | 0.918 | 60.705 | 0.916 | 0.918 | 60.705 | 0.975 | 0.970 | 51.870 | 0.809 | 0.804 | 60.140 | 1.000 | 0.852 | 13.569 | 0.717 | 0.718 | 3.784 | 0.916 | 0.918 | 26.312|
| LP | Train-F1-OCEL | Test-F1-OCEL | RT-OCEL | Train-F1-CELOE | Test-F1-CELOE | RT-CELOE | Train-F1-Evo | Test-F1-Evo | RT-Evo | Train-F1-DRILL | Test-F1-DRILL | RT-DRILL | Train-F1-TDL | Test-F1-TDL | RT-TDL | Train-F1-NCES | Test-F1-NCES | RT-NCES | Train-F1-NCES2 | Test-F1-NCES2 | RT-NCES2 | Train-F1-ROCES | Test-F1-ROCES | RT-ROCES | Train-F1-CLIP | Test-F1-CLIP | RT-CLIP |
|:---------|--------------:|-------------:|--------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|----------------:|---------------:|----------:|-----------------:|----------------:|-----------:|-----------------:|----------------:|-----------:|--------------:|-------------:|--------:|
| NotKnown | 0.916 | 0.918 | 60.705 | 0.916 | 0.918 | 60.705 | 0.975 | 0.970 | 51.870 | 0.809 | 0.804 | 60.140 | 1.000 | 0.852 | 13.569 | 0.704 | 0.704 | 2.605 | 0.704 | 0.704 | 1.841 | 0.704 | 0.704 | 1.711 | 0.916 | 0.918 | 26.312|



### 10-Fold Cross Validation Carcinogenesis Benchmark Results
```shell
python examples/concept_learning_cv_evaluation.py --kb ./KGs/Carcinogenesis/carcinogenesis.owl --lps ./LPs/Carcinogenesis/lps.json --path_of_nces_embeddings ./NCESData/carcinogenesis/embeddings/ConEx_entity_embeddings.csv --path_of_clip_embeddings ./CLIPData/carcinogenesis/embeddings/ConEx_entity_embeddings.csv --max_runtime 60 --report carcinogenesis_results.csv
```
| LP | Train-F1-OCEL | Test-F1-OCEL | RT-OCEL | Train-F1-CELOE | Test-F1-CELOE | RT-CELOE | Train-F1-Evo | Test-F1-Evo | RT-Evo | Train-F1-DRILL | Test-F1-DRILL | RT-DRILL | Train-F1-TDL | Test-F1-TDL | RT-TDL | Train-F1-NCES | Test-F1-NCES | RT-NCES | Train-F1-CLIP | Test-F1-CLIP | RT-CLIP |
|:---------|--------------:|-------------:|--------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|--------------:|-------------:|--------:|--------------:|-------------:|--------:|
| NOTKNOWN | 0.737 | 0.711 | 62.048 | 0.740 | 0.701 | 62.048 | 0.822 | 0.628 | 64.508 | 0.740 | 0.707 | 60.120 | 1.000 | 0.616 | 5.196 | 0.705 | 0.704 | 4.157 | 0.740 | 0.701 | 48.475|

| LP | Train-F1-OCEL | Test-F1-OCEL | RT-OCEL | Train-F1-CELOE | Test-F1-CELOE | RT-CELOE | Train-F1-Evo | Test-F1-Evo | RT-Evo | Train-F1-DRILL | Test-F1-DRILL | RT-DRILL | Train-F1-TDL | Test-F1-TDL | RT-TDL | Train-F1-NCES | Test-F1-NCES | RT-NCES | Train-F1-NCES2 | Test-F1-NCES2 | RT-NCES2 | Train-F1-ROCES | Test-F1-ROCES | RT-ROCES | Train-F1-CLIP | Test-F1-CLIP | RT-CLIP |
|:---------|--------------:|-------------:|--------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|----------------:|---------------:|----------:|-----------------:|----------------:|-----------:|-----------------:|----------------:|-----------:|--------------:|-------------:|--------:|
| NOTKNOWN | 0.737 | 0.711 | 62.048 | 0.740 | 0.701 | 62.048 | 0.822 | 0.628 | 64.508 | 0.740 | 0.707 | 60.120 | 1.000 | 0.616 | 5.196 | 0.709 | 0.709 | 2.718 | 0.705 | 0.704 | 1.912 | 0.705 | 0.704 | 1.774 | 0.740 | 0.701 | 48.475|


### Benchmark Results on DBpedia. Results are based on the training examples only
Expand Down Expand Up @@ -335,6 +337,22 @@ pytest -p no:warnings -x # Running 76 tests takes ~ 17 mins
Currently, we are working on our manuscript describing our framework.
If you find our work useful in your research, please consider citing the respective paper:
```
# ROCES
@inproceedings{kouagou2024roces,
title = {ROCES: Robust Class Expression Synthesis in Description Logics via Iterative Sampling},
author = {Kouagou, N'Dah Jean and Heindorf, Stefan and Demir, Caglar and Ngonga Ngomo, Axel-Cyrille},
booktitle = {Proceedings of the Thirty-Third International Joint Conference on
Artificial Intelligence, {IJCAI-24}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Kate Larson},
pages = {4335--4343},
year = {2024},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2024/479},
url = {https://doi.org/10.24963/ijcai.2024/479},
}
# DRILL
@inproceedings{demir2023drill,
author = {Demir, Caglar and Ngomo, Axel-Cyrille Ngonga},
Expand Down
Loading

0 comments on commit 1f36cd0

Please sign in to comment.