This repository has been archived by the owner on Jan 10, 2023. It is now read-only.
Alias transfer #326
Merged
Alias transfer #326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rahul1980
approved these changes
Jan 21, 2019
sling/nlp/wiki/aliases.cc
Outdated
| @@ -238,7 +258,7 @@ class ProfileAliasReducer : public task::Reducer { | |||
| merged.Add(n_alias_, a.Create()); | |||
| } | |||
|
|
|||
| // Output alias profile. | |||
| // Output selected aliased. | |||
sling/nlp/wiki/aliases.cc
Outdated
| @@ -66,18 +66,34 @@ class ProfileAliasExtractor : public task::FrameProcessor { | |||
| AddAlias(&a, alias.GetHandle(n_name_), SRC_WIKIDATA_FOREIGN, | |||
| alias.GetHandle(n_lang_), alias.GetInt(n_count_)); | |||
| } | |||
| } else if (s.name == n_native_name_ || s.name == n_native_label_) { | |||
| } else if (s.name == n_native_name_ || | |||
| s.name == n_native_label_) { | |||
There was a problem hiding this comment.
Can fit on previous line?
| // Prune phrase table by transfering unreliable aliases to reliable | ||
| // aliases for related items. | ||
| if (transfer_aliases_) { | ||
| LOG(INFO) << "Transfer aliases"; |
There was a problem hiding this comment.
Do you still need the LOG(INFO) statements in this method?
There was a problem hiding this comment.
I use it for verifying that alias transfer is enabled and it is also useful for timing the alias transfer.
sling/nlp/kb/phrase-table-builder.cc
Outdated
| bool reliable() const { return count_and_flags & (1 << 31); } | ||
|
|
||
| // Phrase form. | ||
| int form() const { return (count_and_flags << 29) & 3; } |
There was a problem hiding this comment.
shouldn't this be (count_and_flags >> 29) & 3 ?
There was a problem hiding this comment.
Oops. That's a bug. Good catch!
ringgaard
commented
Jan 22, 2019
| // Prune phrase table by transfering unreliable aliases to reliable | ||
| // aliases for related items. | ||
| if (transfer_aliases_) { | ||
| LOG(INFO) << "Transfer aliases"; |
There was a problem hiding this comment.
I use it for verifying that alias transfer is enabled and it is also useful for timing the alias transfer.
sling/nlp/kb/phrase-table-builder.cc
Outdated
| bool reliable() const { return count_and_flags & (1 << 31); } | ||
|
|
||
| // Phrase form. | ||
| int form() const { return (count_and_flags << 29) & 3; } |
There was a problem hiding this comment.
Oops. That's a bug. Good catch!
sling/nlp/wiki/aliases.cc
Outdated
| @@ -66,18 +66,34 @@ class ProfileAliasExtractor : public task::FrameProcessor { | |||
| AddAlias(&a, alias.GetHandle(n_name_), SRC_WIKIDATA_FOREIGN, | |||
| alias.GetHandle(n_lang_), alias.GetInt(n_count_)); | |||
| } | |||
| } else if (s.name == n_native_name_ || s.name == n_native_label_) { | |||
| } else if (s.name == n_native_name_ || | |||
| s.name == n_native_label_) { | |||
sling/nlp/wiki/aliases.cc
Outdated
| @@ -238,7 +258,7 @@ class ProfileAliasReducer : public task::Reducer { | |||
| merged.Add(n_alias_, a.Create()); | |||
| } | |||
|
|
|||
| // Output alias profile. | |||
| // Output selected aliased. | |||
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We use anchors, redirects, and wikipedia titles as alias sources for building the phrase table, but these are all noisy sources. I have implemented "alias transfer" in the phrase table builder to clean up some of these noisy aliases. The basic idea is that an item cannot have the same name as another item that it is related to, except for a small list of exceptions (e.g. named after, different from etc.). The aliases are divided in reliable and unreliable aliases based on the source, and if an item has an unreliable alias and is related to another item which has the same alias from a reliable source, the alias count is transferred to the target.
The alias transfer procedure cleans up many cases of noisy aliases. For instance "botanist" is remove from "botany" because
[botanist (Q2374149) field of this occupation (P425): botany (Q441)].Wikidata has finer granularity that any individual Wikipedia, so the alias transfer also fixes many cases where a "sub-item" has a redirect to a broader item, e.g. villages that are not in Wikipedia redirects to the county, taxon redirects to parent taxon etc.
Removing all punctuation for alias normalization introduces too many false matches, so I have changed the default normalization for the phrase table so only dashes and periods are normalized.
Other minor changes: