Skip to content

Commit

Permalink
Evaluation (#12)
Browse files Browse the repository at this point in the history 
commit 6c784ee55a9ffe3e80a26a35aac3d0d14d9e289c
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 22:15:59 2024 +0200

    ✨ Add script to merge json output from mixtal/mistral

commit 1ae2d4f9eba8616c717eed5310bf33ba2aeca443
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 16:33:07 2024 +0200

    🗃️ Remove current version of db

commit 568eb1a01f0f117083a8b1fa5b36681e9aeca3e5
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 16:30:46 2024 +0200

    🙈 Fix gitignore file

    Clean up, generalize file ignoring

commit fcdc49c3bab7cb7ed3e58a7d1517b0c616264f64
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 16:30:05 2024 +0200

    🔥 Remove surplus versions of test/dev sys output

commit d58f15a8c1a713eccee06cbaea1d732f8637ab94
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 15:57:28 2024 +0200

    ♻️ Refactor script

commit 115121006aca09a36adbdf7d82b47a975085dc54
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 15:21:26 2024 +0200

    📝 Add README on raw dir for system output

commit 7c1261e3353040f8902091fceb28ea126c73db51
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 15:10:52 2024 +0200

    🚚 Move eval results (first iteration) into "nlp4climate" dir

commit 9abdf74cb49054f3890d8b801af6fcad36630d28
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 15:09:58 2024 +0200

    🔥 Remove parsed files with missing events

commit ba5194e0bc0cc8fe105dd2320e8a235fa3a9d1b0
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 15:07:52 2024 +0200

    🔥 Remove invalid gold dev/test sets

commit e6c1e74fc28bd031387c15a9a1900bc6d7e72261
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 15:06:01 2024 +0200

    🚚 Structure system outputs in clear dir breakdown

commit ea72a5f8c5aa100ea72f7e07f2c3524d63ac05e2
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 14:39:10 2024 +0200

    💬 Remove outdated prompts

commit 9ea9b21ada0d3e993ac08fa717cb471d27ff89d1
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 14:33:07 2024 +0200

    🔥 Remove old eval results (no longer valid)

commit b954d5968fe61888eb7abce9638d94ea21468918
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 27 14:30:16 2024 +0200

    ⚗️ Add eval results for nlp4climate paper

commit 86f5bbb8075592f86e4bd485c49fad22deddbafb
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 20:19:04 2024 +0200

    🚚 Move into dirs (dev/test)

commit 641dd1da2d87cd4b6545ac145cc1597a3be78d6a
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 20:13:28 2024 +0200

    🙈 Ignore some files (temp)

commit 8410b42a40b946b4ff75246d4655966f37ba2081
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 20:13:03 2024 +0200

    ✨ Evaluate by source of event (artemis/wikipedia)

commit 2926174c0c4ad336b888f2b40a7c1f24c4e8e330
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 20:11:05 2024 +0200

    🩹 Add source col to mistral/mixtral models

commit 195edbe88c5216bd809014f55daaeeb158746aab
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 20:10:40 2024 +0200

    ⚗️ Update gold

commit 559fd5a73bb2321a2914932d39137933cec40373
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 20:10:18 2024 +0200

    ⚗️ Add evaluation results (wikipedia, artemis, both)

commit 45491d1766e800d7acf448b71016984f4d5a0d56
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 16:44:09 2024 +0200

    ⚗️ Update gold test

commit 9facc97db7ebd40fdbe07ffe0066e1935df3cfde
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 16:40:16 2024 +0200

    ⚗️ Add raw system output for the dev and test sets

commit efaa05d53999e7bb6ec211cb7a6fb00607599a76
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 16:39:40 2024 +0200

    ⚗️ Add parsed dev and test system output files

commit 05d22571cbba55703038129cddf464889bd36b59
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 16:37:58 2024 +0200

    🚚 Move evaluation results that may contain errors

commit 13f0893b5aac682b7bf8654adeb231bc354ba9a8
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 13:46:18 2024 +0200

    🩹 Fix upper_case multiplier (7 -> 9)

commit a7687bf32de7a8d403a6f7dfe9e70fc710a2204d
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 13:19:08 2024 +0200

    🧑‍💻 Fix names (not "dev", but "sys" output)

commit 335608be9337fd56373269aeeb283c363f5e65aa
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 13:18:30 2024 +0200

    🏗️ Evaluate None matches as zero

    If the system extracts "NULL" and our gold set says "NULL", return a score of 0 (aka fully correct)

commit 0b8925b7f503e99f06b994455a4a6f3a60d001ef
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 13:07:17 2024 +0200

    🚚 Update filename for consistency

commit b1f472033c2fca3bc22aec747b48c862b705cff3
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 16 13:04:45 2024 +0200

    🎨 Order results by weight keys

commit f993f9e97fe5b639a8517e4c0ebe34dd28848c2b
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 16:01:21 2024 +0200

    ♻️ Remove unnecessary normalization

commit 33e45dd0e6a48151ddb33e98c440cc94207b162f
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 16:00:57 2024 +0200

    📦️ Add output files

commit a59975abc1d71405649e5a6c7a5e6b8a92b1ca68
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 15:59:48 2024 +0200

    🔥 Remove dead files

commit fbdc5ae5a98de0ee1e951fc2358b036d2d12d49e
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 15:58:18 2024 +0200

    📦️ Add eval results for all models

commit e96f947a6e747c78f3cf430dbdd4c2523391298b
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 15:49:34 2024 +0200

    📦️ Fix column type for parquet

commit b556c809a98127a86f76c080478de432ae65bb68
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:29:42 2024 +0200

    🚨 Fix lint formatting

commit f886fd574f5c8cb6eb719bba5e8cf71d2deb4fe3
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:28:52 2024 +0200

    ♻️ Refactor function name that removes non-target columns

commit 75887b399ef866c27ce93ff96c335c6e54773a7b
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:27:21 2024 +0200

    📦️ Add raw test set for gpt4

commit 5a522a23357839b8f4fc12524fc61c48c5090d5e
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:24:00 2024 +0200

    💩 Add mixtral merging script (should be removed or turned into a python file later)

commit b047f5abb8830a4bf52bd39341be49b201541c73
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:22:28 2024 +0200

    🙈 Ignore excel sheets

commit 44142701e5d6d9fecbcce53d6d5ad22d6109ca86
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:19:51 2024 +0200

    ⚗️ Add gold excel sheets (derived from the manual validation table)

commit 904599121c91eed8d7edb336c4231e4351df8b66
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:19:07 2024 +0200

    📦️ Add parsed and normalized output files

commit 372d0199c5ec3d8619c9151eae6567735f4e6f81
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:14:57 2024 +0200

    ⚗️ Add gold for dev and test sets

commit e7e8355dd139b2fc8a8783d48b29d92cb6e0e96e
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:12:55 2024 +0200

    🙈 Ignore all mixtral and mistral dirs

commit 46ba814d4c5efb1c350c14638e0eb1ccebd5cd80
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:12:13 2024 +0200

    🚑️ Hotfix for unwanted columns

commit 31493dc1ec2b84c35f7cf666b7fa6dead6e23829
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Wed May 15 12:09:53 2024 +0200

    ✨ Add automated eval script

commit efd2ca5f65161fd926a53d3cd4cc3a006a18108b
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 14:55:33 2024 +0200

    📦️ Add mistral parsed files and results

commit 0b03efbf3e10b039c401f24125ea801fac68dcad
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 14:55:08 2024 +0200

    🧑‍💻 Move helper functions to utils

commit bc07cef403f1872c251e41cc6d6be2c64b44e63e
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 14:54:18 2024 +0200

    🧑‍💻 Give logs their own class

commit ce64fc52d827eb4142ed0709b2addd00096addcd
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 10:50:46 2024 +0200

    💡 Push latest updates to notebooks

commit 6a35a497dcba9771cc88b6693368989f685f538f
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 10:49:53 2024 +0200

    ⚡️ Remove small locations from country list

commit 2f97bb012b00399a2e09b2237cf6b256ebbd0d2f
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 10:48:42 2024 +0200

    ⚗ Literal eval for dict or list as strings

commit a645b038792376eada0b3d63f87580d096747077
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 10:46:04 2024 +0200

    ⚡️ Strip area name of underscores

commit 147c202c1eb5c91dbecc11c6851bf687c490ddca
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 10:44:50 2024 +0200

    🗃️ Add event id mapping

commit 53b89fd3e2135e54a4200880b7dbcb8c16ac591a
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Mon May 13 10:43:42 2024 +0200

    📦️ Upload parsed files

commit 4e5b7d3d05cbb0a2b80a212f6b0ccfb0cb11c20b
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 23:38:17 2024 +0200

    🙈 Ignore mistral files

commit b7d0330834a747b059ae8b4d4b1bdff3d8d24391
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 23:37:59 2024 +0200

    🧑‍💻 Add option for custom location/country col

commit 2e8c43edbdb38fd5692d76ef44bb639d5164d2f9
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 23:37:22 2024 +0200

    💡 Comment-out code that drops rows with missing essential info

commit ccc3b65a8a27003f01317326826217d0c0624d95
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 23:34:38 2024 +0200

    📦️ Add parquet and csv result files for dev, test, mixtral, and mistral

commit 3b4c2d5b88e0900609b918f469c087de1257f35a
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 16:10:35 2024 +0200

    ♻️ Add funcitonality to select target columns for eval

commit 6f4e94ea53f00e6d7a8716d574fbc91853dc40fa
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 14:02:07 2024 +0200

    📦️ Add output files

commit 536902c84b47633e5e9dca62b07d15db2d65625d
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 14:01:08 2024 +0200

    📦️ Add raw files

commit cbfc20f59e8b3f55c6f2c3454aa51634deafc483
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 12:59:23 2024 +0200

    🧑‍💻 Add draft notebook for evaluation (test set as example)

commit b1778c329c139244ff7f359a11b8e2bef41e31f5
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 12:58:52 2024 +0200

    ⚡️ Remove sequence normalization in normalizer

    Temp solution, clean later

commit 0adbebbc0a4f9db8a2ec69744dc5bb49660a29c0
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 12:56:07 2024 +0200

    💬 Update comparer and test

commit 9b6a33eeb6c8a6e28dec86cacda4e912b85cf719
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 12:54:55 2024 +0200

    ⚡️ Normalize more "none" phrases to 0

commit cf268caadc47023ff763ce3f332376b9bc09b009
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 12:53:59 2024 +0200

    🗃️ Only include cardinals if location != country

commit 0ee86c12510034b2dc1f71185e9204ec8a97d6e5
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 12:53:19 2024 +0200

    🏷️ Fix column types

commit 5ed0e816a1a95f99f0bf16c3eece5618f7e32737
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 12:52:15 2024 +0200

    📌 Update lock and pin deps

    Add numpy as dev dep

commit 49bd5cdc3d57579bedea8dffac77c0d1c233f94a
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Sun May 12 12:51:01 2024 +0200

    🔥 Remove old parsed files

commit b2312452662960c6611d7bc3edf4e6ac99f4fe39
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 9 21:07:06 2024 +0200

    🗃️ Add latest parsed events for eval

commit e445534756dab2d0405243db7105de4e172b3838
Author: Shorouq <shorouq.zahra@ri.se>
Date:   Thu May 9 20:00:43 2024 +0200

    🗃️ Merge json for deaths, damage, specific deaths/damage

Co-authored-by: Shorouq <shorouq.zahra@ri.se>
  • Loading branch information
@i-be-snek
i-be-snek and i-be-snek authored May 27, 2024
1 parent 501df99 commit 9ccff02
Show file tree
Hide file tree
Showing 97 changed files with 57,739 additions and 20,506 deletions.
15 changes: 9 additions & 6 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
# ignore dev files
# ignore raw files
Database/raw/*/*/dev
Database/raw/*/*/test
results
.env
Database/raw/*.xlsx
Database/output/*.csv
Database/output/Ni/*.csv
Database/output/dev/*

# ignore notebooks
**.ipynb

# ignore excel files
**.xlsx

# ignore pycache
**/__pycache__
Expand Down
21 changes: 0 additions & 21 deletions Database/create_db.py
View file

This file was deleted.

18 changes: 18 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/gpt4-dev_all_100_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Coverage": 0.0,
"Weighted_Score": 0.11455204011706782,
"Main_Event": 0.08,
"Event_Name": 0.07,
"Total_Deaths_Min": 0.0640545422556355,
"Total_Deaths_Max": 0.060924784205546444,
"Total_Damage_Min": 0.19119539392735532,
"Total_Damage_Max": 0.18726556936595182,
"Total_Damage_Units": 0.3,
"Start_Date_Day": 0.10315022456010826,
"Start_Date_Month": 0.04923076923076923,
"Start_Date_Year": 0.020020028900900665,
"End_Date_Day": 0.12454699606438738,
"End_Date_Month": 0.0584384037015616,
"End_Date_Year": 0.030032414714499328,
"Country_Norm": 0.3348694347122339
}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/gpt4-dev_all_100_results.csv
View file
Git LFS file not shown
18 changes: 18 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/gpt4-dev_artemis_16_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Coverage": 0.0,
"Weighted_Score": 0.2735838212850395,
"Main_Event": 0.0,
"Event_Name": 0.0,
"Total_Deaths_Min": 0.1875,
"Total_Deaths_Max": 0.1875,
"Total_Damage_Min": 0.6166666666666667,
"Total_Damage_Max": 0.5921052631578947,
"Total_Damage_Units": 0.75,
"Start_Date_Day": 0.33861664820967147,
"Start_Date_Month": 0.0625,
"Start_Date_Year": 0.0625,
"End_Date_Day": 0.3925694444444444,
"End_Date_Month": 0.11195054945054944,
"End_Date_Year": 0.06251552023839087,
"Country_Norm": 0.4657494058229352
}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/gpt4-dev_artemis_16_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions Database/evaluation_results/nlp4climate/dev/gpt4-dev_wikipedia_84_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.08426027227554941, "Main_Event": 0.09523809523809523, "Event_Name": 0.08333333333333333, "Total_Deaths_Min": 0.040541121732899414, "Total_Deaths_Max": 0.036815219292317194, "Total_Damage_Min": 0.11015324673891506, "Total_Damage_Max": 0.11015324673891506, "Total_Damage_Units": 0.21428571428571427, "Start_Date_Day": 0.05829947719828672, "Start_Date_Month": 0.046703296703296704, "Start_Date_Year": 0.011928605834405552, "End_Date_Day": 0.07349510113485269, "End_Date_Month": 0.04824561403508772, "End_Date_Year": 0.023845156519472367, "Country_Norm": 0.3099399164054337}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/gpt4-dev_wikipedia_84_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions Database/evaluation_results/nlp4climate/dev/mistral-7b-instruct-dev_all_100_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.49741124670592785, "Main_Event": 0.07, "Event_Name": 0.91, "Total_Deaths_Min": 0.26132014455529706, "Total_Deaths_Max": 0.26865066141594196, "Total_Damage_Min": 0.49087778518025016, "Total_Damage_Max": 0.4802923976608187, "Total_Damage_Units": 0.41, "Start_Date_Day": 0.7528253968253967, "Start_Date_Month": 0.75, "Start_Date_Year": 0.74, "End_Date_Day": 0.7641374269005848, "End_Date_Month": 0.76, "End_Date_Year": 0.7500024869435464, "Country_Norm": 0.4656511544011544}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/mistral-7b-instruct-dev_all_100_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions ...se/evaluation_results/nlp4climate/dev/mistral-7b-instruct-dev_artemis_16_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.36449596834347775, "Main_Event": 0.0625, "Event_Name": 0.5, "Total_Deaths_Min": 0.25, "Total_Deaths_Max": 0.25, "Total_Damage_Min": 0.3041666666666667, "Total_Damage_Max": 0.27960526315789475, "Total_Damage_Units": 0.75, "Start_Date_Day": 0.39087301587301587, "Start_Date_Month": 0.375, "Start_Date_Year": 0.375, "End_Date_Day": 0.4600694444444444, "End_Date_Month": 0.4375, "End_Date_Year": 0.4375, "Country_Norm": 0.7307291666666667}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/mistral-7b-instruct-dev_artemis_16_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions .../evaluation_results/nlp4climate/dev/mistral-7b-instruct-dev_wikipedia_84_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.5227284425844898, "Main_Event": 0.07142857142857142, "Event_Name": 0.9880952380952381, "Total_Deaths_Min": 0.26347636256582985, "Total_Deaths_Max": 0.2722031683523119, "Total_Damage_Min": 0.5264418077542662, "Total_Damage_Max": 0.5185185185185185, "Total_Damage_Units": 0.34523809523809523, "Start_Date_Day": 0.8217687074829931, "Start_Date_Month": 0.8214285714285714, "Start_Date_Year": 0.8095238095238095, "End_Date_Day": 0.8220551378446116, "End_Date_Month": 0.8214285714285714, "End_Date_Year": 0.8095267701708886, "Country_Norm": 0.4151601044458187}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/mistral-7b-instruct-dev_wikipedia_84_results.csv
View file
Git LFS file not shown
18 changes: 18 additions & 0 deletions ...ase/evaluation_results/nlp4climate/dev/mixtral-8x7b-instruct-dev_all_100_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Coverage": 0.0,
"Weighted_Score": 0.23242246881152565,
"Main_Event": 0.08,
"Event_Name": 0.91,
"Total_Deaths_Min": 0.23885661847612297,
"Total_Deaths_Max": 0.23592654191568233,
"Total_Damage_Min": 0.33440763504833526,
"Total_Damage_Max": 0.33440763504833526,
"Total_Damage_Units": 0.38,
"Start_Date_Day": 0.20779663808450635,
"Start_Date_Month": 0.15052631578947367,
"Start_Date_Year": 0.13001991498193471,
"End_Date_Day": 0.2874742780374359,
"End_Date_Month": 0.20531235328139352,
"End_Date_Year": 0.19022311745521212,
"Country_Norm": 0.47896351524292696
}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/mixtral-8x7b-instruct-dev_all_100_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions .../evaluation_results/nlp4climate/dev/mixtral-8x7b-instruct-dev_artemis_16_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.42638613570055334, "Main_Event": 0.125, "Event_Name": 0.5, "Total_Deaths_Min": 0.25, "Total_Deaths_Max": 0.25, "Total_Damage_Min": 0.6875, "Total_Damage_Max": 0.6875, "Total_Damage_Units": 0.8125, "Start_Date_Day": 0.4546880767811, "Start_Date_Month": 0.3125, "Start_Date_Year": 0.3125, "End_Date_Day": 0.58875, "End_Date_Month": 0.4423076923076923, "End_Date_Year": 0.4375, "Country_Norm": 0.6086601307189543}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/dev/mixtral-8x7b-instruct-dev_artemis_16_results.csv
View file
Git LFS file not shown
18 changes: 18 additions & 0 deletions ...valuation_results/nlp4climate/dev/mixtral-8x7b-instruct-dev_wikipedia_84_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Coverage": 0.0,
"Weighted_Score": 0.19547700845171084,
"Main_Event": 0.07142857142857142,
"Event_Name": 0.9880952380952381,
"Total_Deaths_Min": 0.2367340696144321,
"Total_Deaths_Max": 0.23324588323295514,
"Total_Damage_Min": 0.26715194648611335,
"Total_Damage_Max": 0.26715194648611335,
"Total_Damage_Units": 0.2976190476190476,
"Start_Date_Day": 0.1607696973803933,
"Start_Date_Month": 0.11967418546365914,
"Start_Date_Year": 0.09526180354992225,
"End_Date_Day": 0.23008842623504278,
"End_Date_Month": 0.16017038394305091,
"End_Date_Year": 0.1431227588752525,
"Country_Norm": 0.454259398009398
}
3 changes: 3 additions & 0 deletions ...ase/evaluation_results/nlp4climate/dev/mixtral-8x7b-instruct-dev_wikipedia_84_results.csv
View file
Git LFS file not shown
18 changes: 18 additions & 0 deletions Database/evaluation_results/nlp4climate/test/gpt4-test_all_170_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Coverage": 0.0,
"Weighted_Score": 0.08178022048535592,
"Main_Event": 0.10588235294117647,
"Event_Name": 0.03529411764705882,
"Total_Deaths_Min": 0.04556058520253607,
"Total_Deaths_Max": 0.046152826349527803,
"Total_Damage_Min": 0.15056886714649706,
"Total_Damage_Max": 0.15056886714649706,
"Total_Damage_Units": 0.12941176470588237,
"Start_Date_Day": 0.047462286031857384,
"Start_Date_Month": 0.04313725490196079,
"Start_Date_Year": 0.041176470588235294,
"End_Date_Day": 0.03924939437602325,
"End_Date_Month": 0.026722689075630253,
"End_Date_Year": 0.02353380268617458,
"Country_Norm": 0.2954959256429845
}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/test/gpt4-test_all_170_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions Database/evaluation_results/nlp4climate/test/gpt4-test_artemis_12_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.2249114845809375, "Main_Event": 0.08333333333333333, "Event_Name": 0.0, "Total_Deaths_Min": 0.16666666666666666, "Total_Deaths_Max": 0.16666666666666666, "Total_Damage_Min": 0.8333333333333334, "Total_Damage_Max": 0.8333333333333334, "Total_Damage_Units": 0.8333333333333334, "Start_Date_Day": 0.0, "Start_Date_Month": 0.0, "Start_Date_Year": 0.0, "End_Date_Day": 0.004154002026342452, "End_Date_Month": 0.011904761904761904, "End_Date_Year": 0.0, "Country_Norm": 0.21603535353535352}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/test/gpt4-test_artemis_12_results.csv
View file
Git LFS file not shown
18 changes: 18 additions & 0 deletions Database/evaluation_results/nlp4climate/test/gpt4-test_wikipedia_158_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Coverage": 0.0,
"Weighted_Score": 0.07090949156670416,
"Main_Event": 0.10759493670886076,
"Event_Name": 0.0379746835443038,
"Total_Deaths_Min": 0.03636265496475401,
"Total_Deaths_Max": 0.03699987645202357,
"Total_Damage_Min": 0.09871333806901585,
"Total_Damage_Max": 0.09871333806901585,
"Total_Damage_Units": 0.0759493670886076,
"Start_Date_Day": 0.05106701661655541,
"Start_Date_Month": 0.046413502109704644,
"Start_Date_Year": 0.04430379746835443,
"End_Date_Day": 0.041914867212707874,
"End_Date_Month": 0.027848101265822787,
"End_Date_Year": 0.025321180105377712,
"Country_Norm": 0.30153090580305775
}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/test/gpt4-test_wikipedia_158_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions ...ase/evaluation_results/nlp4climate/test/mistral-7b-instruct-test_all_170_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.503042979864537, "Main_Event": 0.08823529411764706, "Event_Name": 0.9470588235294117, "Total_Deaths_Min": 0.19931397568201362, "Total_Deaths_Max": 0.2092446618687604, "Total_Damage_Min": 0.6112983489618762, "Total_Damage_Max": 0.5996333615705598, "Total_Damage_Units": 0.29411764705882354, "Start_Date_Day": 0.762139662882697, "Start_Date_Month": 0.753250773993808, "Start_Date_Year": 0.7530681767245887, "End_Date_Day": 0.7768207282913167, "End_Date_Month": 0.7726050420168068, "End_Date_Year": 0.7705896952973296, "Country_Norm": 0.45228434963729086}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/test/mistral-7b-instruct-test_all_170_results.csv
View file
Git LFS file not shown
18 changes: 18 additions & 0 deletions .../evaluation_results/nlp4climate/test/mistral-7b-instruct-test_artemis_12_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Coverage": 0.0,
"Weighted_Score": 0.27984221466364323,
"Main_Event": 0.08333333333333333,
"Event_Name": 0.25,
"Total_Deaths_Min": 0.16666666666666666,
"Total_Deaths_Max": 0.16666666666666666,
"Total_Damage_Min": 0.4166666666666667,
"Total_Damage_Max": 0.4166666666666667,
"Total_Damage_Units": 1.0,
"Start_Date_Day": 0.0925925925925926,
"Start_Date_Month": 0.08333333333333333,
"Start_Date_Year": 0.08333333333333333,
"End_Date_Day": 0.25,
"End_Date_Month": 0.2619047619047619,
"End_Date_Year": 0.25,
"Country_Norm": 0.6466269841269842
}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/test/mistral-7b-instruct-test_artemis_12_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions ...aluation_results/nlp4climate/test/mistral-7b-instruct-test_wikipedia_158_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.5199949367152378, "Main_Event": 0.08860759493670886, "Event_Name": 1.0, "Total_Deaths_Min": 0.2017935181388754, "Total_Deaths_Max": 0.21247843365626118, "Total_Damage_Min": 0.6260805020475884, "Total_Damage_Max": 0.6135295662468049, "Total_Damage_Units": 0.24050632911392406, "Start_Date_Day": 0.8129913391072618, "Start_Date_Month": 0.8041305796135909, "Start_Date_Year": 0.8039341141973423, "End_Date_Day": 0.8168324291742014, "End_Date_Month": 0.811392405063291, "End_Date_Year": 0.8101281531680128, "Country_Norm": 0.437524149549466}
3 changes: 3 additions & 0 deletions ...se/evaluation_results/nlp4climate/test/mistral-7b-instruct-test_wikipedia_158_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions ...e/evaluation_results/nlp4climate/test/mixtral-8x7b-instruct-test_all_170_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.23495496479778205, "Main_Event": 0.1, "Event_Name": 0.9470588235294117, "Total_Deaths_Min": 0.18778216614802207, "Total_Deaths_Max": 0.18345104828921494, "Total_Damage_Min": 0.4537675508169044, "Total_Damage_Max": 0.4537675508169044, "Total_Damage_Units": 0.3941176470588235, "Start_Date_Day": 0.1671475993160761, "Start_Date_Month": 0.15047913902403068, "Start_Date_Year": 0.1411764705882353, "End_Date_Day": 0.2265569896562948, "End_Date_Month": 0.1961344537815126, "End_Date_Year": 0.18851703020106916, "Country_Norm": 0.44647186147186146}
3 changes: 3 additions & 0 deletions Database/evaluation_results/nlp4climate/test/mixtral-8x7b-instruct-test_all_170_results.csv
View file
Git LFS file not shown
18 changes: 18 additions & 0 deletions ...valuation_results/nlp4climate/test/mixtral-8x7b-instruct-test_artemis_12_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Coverage": 0.0,
"Weighted_Score": 0.2587790813123743,
"Main_Event": 0.08333333333333333,
"Event_Name": 0.25,
"Total_Deaths_Min": 0.16666666666666666,
"Total_Deaths_Max": 0.16666666666666666,
"Total_Damage_Min": 0.3333333333333333,
"Total_Damage_Max": 0.3333333333333333,
"Total_Damage_Units": 0.75,
"Start_Date_Day": 0.16807909604519775,
"Start_Date_Month": 0.16666666666666666,
"Start_Date_Year": 0.16666666666666666,
"End_Date_Day": 0.25,
"End_Date_Month": 0.2619047619047619,
"End_Date_Year": 0.25,
"Country_Norm": 0.5262566137566139
}
3 changes: 3 additions & 0 deletions ...ase/evaluation_results/nlp4climate/test/mixtral-8x7b-instruct-test_artemis_12_results.csv
View file
Git LFS file not shown
1 change: 1 addition & 0 deletions ...uation_results/nlp4climate/test/mixtral-8x7b-instruct-test_wikipedia_158_avg_results.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"Coverage": 0.0, "Weighted_Score": 0.23314553822705353, "Main_Event": 0.10126582278481013, "Event_Name": 1.0, "Total_Deaths_Min": 0.1893858749693908, "Total_Deaths_Max": 0.18472581145042113, "Total_Damage_Min": 0.46291445341059323, "Total_Damage_Max": 0.46291445341059323, "Total_Damage_Units": 0.3670886075949367, "Start_Date_Day": 0.1670768527290542, "Start_Date_Month": 0.14924970654484312, "Start_Date_Year": 0.13924050632911392, "End_Date_Day": 0.22477650785803874, "End_Date_Month": 0.1911392405063291, "End_Date_Year": 0.1838474375581124, "Country_Norm": 0.4404122600325132}
3 changes: 3 additions & 0 deletions .../evaluation_results/nlp4climate/test/mixtral-8x7b-instruct-test_wikipedia_158_results.csv
View file
Git LFS file not shown
3 changes: 3 additions & 0 deletions Database/gold/gold_dev_20240515.parquet
View file
Git LFS file not shown
3 changes: 3 additions & 0 deletions Database/gold/gold_test_20240515.parquet
View file
Git LFS file not shown
Loading

0 comments on commit 9ccff02

Please sign in to comment.