[Logger Framework] Log Losses #1934
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
17 tasks
bfineran
reviewed
Jan 3, 2024
Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper
a32ef0f to
b71e119
Compare
0db7679 to
e6002e2
Compare
Add a stateless log_ready method
bfineran
approved these changes
Jan 12, 2024
rahul-tuli
added a commit
that referenced
this pull request
Jan 19, 2024
* parent 5007b8c author Rahul Tuli <rahul@neuralmagic.com> 1702308513 -0500 committer rahul-tuli <rahul@neuralmagic.com> 1704999325 -0500 [Feature] Add Model Level Logs to new Framework (#1783) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests `[Feature][Logger Framework]` Add convenience methods for logging strings (#1855) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests * Adds convenience methods to logger manager to better conform to python's `logging.Logger` [Feature] Info logs to console and debug logs to file (#1861) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests * Adds convenience methods to logger manager to better conform to python's `logging.Logger` * Move FileHandler creation above StreamHandler creation * Remove missed comment [Move] logger into it's own package (#1924) improves structure and separation of concerns Add local logs directory to .gitignore (#1925) [Add] frequency_manager to check log "readiness" (#1927) * Add frequency_manager to check log "readiness" * Refactor condition to be more readable [Logger Framework][Add] mode and frequency type to FrequencyManager (#1930) * Add mode and frequency type to FrequencyManager * move tests cases near the test * Style * [Use] frequency manager in logger (#1931) * Make log_frequency a property of FrequencyManager Add FrequencyManager to LoggerManager Add log_wriiten and model_updated to LoggerManager * [Set] [OLD MODIFIERS] _last_log_epoch to None * Update old modifiers to use named arguments to log_ready * [Update][Rename] * `PossibleLoggingMode` --> `LoggingModeType` * `PossibleFrequencyType` --> `FrequencyType` * Fix log format when step is None (#1933) * [Add] Wrap System and Metric Logging into it's own classes (#1932) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * [Logger Refactor] Session based Logs (#1920) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * Add session level logs * [Logger Framework] Log Losses (#1934) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * Add logs for loss * Add test * Use Sparsification Group logger (#1936) * Save last log step Add a stateless log_ready method * log only when loss is not None * Add tb style logging functions (#1919) * Add wandb log function (#1918) * [Test Update] Fix failing tests after adding back old logger (#1963) * [Logger Refactor] Add timer in logger manager (#1967) * Add timer in logger manager * Address review comments * update docstring * fix argument
rahul-tuli
added a commit
that referenced
this pull request
Jan 19, 2024
* parent 5007b8c author Rahul Tuli <rahul@neuralmagic.com> 1702308513 -0500 committer rahul-tuli <rahul@neuralmagic.com> 1704999325 -0500 [Feature] Add Model Level Logs to new Framework (#1783) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests `[Feature][Logger Framework]` Add convenience methods for logging strings (#1855) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests * Adds convenience methods to logger manager to better conform to python's `logging.Logger` [Feature] Info logs to console and debug logs to file (#1861) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests * Adds convenience methods to logger manager to better conform to python's `logging.Logger` * Move FileHandler creation above StreamHandler creation * Remove missed comment [Move] logger into it's own package (#1924) improves structure and separation of concerns Add local logs directory to .gitignore (#1925) [Add] frequency_manager to check log "readiness" (#1927) * Add frequency_manager to check log "readiness" * Refactor condition to be more readable [Logger Framework][Add] mode and frequency type to FrequencyManager (#1930) * Add mode and frequency type to FrequencyManager * move tests cases near the test * Style * [Use] frequency manager in logger (#1931) * Make log_frequency a property of FrequencyManager Add FrequencyManager to LoggerManager Add log_wriiten and model_updated to LoggerManager * [Set] [OLD MODIFIERS] _last_log_epoch to None * Update old modifiers to use named arguments to log_ready * [Update][Rename] * `PossibleLoggingMode` --> `LoggingModeType` * `PossibleFrequencyType` --> `FrequencyType` * Fix log format when step is None (#1933) * [Add] Wrap System and Metric Logging into it's own classes (#1932) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * [Logger Refactor] Session based Logs (#1920) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * Add session level logs * [Logger Framework] Log Losses (#1934) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * Add logs for loss * Add test * Use Sparsification Group logger (#1936) * Save last log step Add a stateless log_ready method * log only when loss is not None * Add tb style logging functions (#1919) * Add wandb log function (#1918) * [Test Update] Fix failing tests after adding back old logger (#1963) * [Logger Refactor] Add timer in logger manager (#1967) * Add timer in logger manager * Address review comments * update docstring * fix argument
rahul-tuli
added a commit
that referenced
this pull request
Jan 22, 2024
* parent 5007b8c author Rahul Tuli <rahul@neuralmagic.com> 1702308513 -0500 committer rahul-tuli <rahul@neuralmagic.com> 1704999325 -0500 [Feature] Add Model Level Logs to new Framework (#1783) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests `[Feature][Logger Framework]` Add convenience methods for logging strings (#1855) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests * Adds convenience methods to logger manager to better conform to python's `logging.Logger` [Feature] Info logs to console and debug logs to file (#1861) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests * Adds convenience methods to logger manager to better conform to python's `logging.Logger` * Move FileHandler creation above StreamHandler creation * Remove missed comment [Move] logger into it's own package (#1924) improves structure and separation of concerns Add local logs directory to .gitignore (#1925) [Add] frequency_manager to check log "readiness" (#1927) * Add frequency_manager to check log "readiness" * Refactor condition to be more readable [Logger Framework][Add] mode and frequency type to FrequencyManager (#1930) * Add mode and frequency type to FrequencyManager * move tests cases near the test * Style * [Use] frequency manager in logger (#1931) * Make log_frequency a property of FrequencyManager Add FrequencyManager to LoggerManager Add log_wriiten and model_updated to LoggerManager * [Set] [OLD MODIFIERS] _last_log_epoch to None * Update old modifiers to use named arguments to log_ready * [Update][Rename] * `PossibleLoggingMode` --> `LoggingModeType` * `PossibleFrequencyType` --> `FrequencyType` * Fix log format when step is None (#1933) * [Add] Wrap System and Metric Logging into it's own classes (#1932) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * [Logger Refactor] Session based Logs (#1920) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * Add session level logs * [Logger Framework] Log Losses (#1934) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * Add logs for loss * Add test * Use Sparsification Group logger (#1936) * Save last log step Add a stateless log_ready method * log only when loss is not None * Add tb style logging functions (#1919) * Add wandb log function (#1918) * [Test Update] Fix failing tests after adding back old logger (#1963) * [Logger Refactor] Add timer in logger manager (#1967) * Add timer in logger manager * Address review comments * update docstring * fix argument
bfineran
pushed a commit
that referenced
this pull request
Jan 22, 2024
* parent 5007b8c author Rahul Tuli <rahul@neuralmagic.com> 1702308513 -0500 committer rahul-tuli <rahul@neuralmagic.com> 1704999325 -0500 [Feature] Add Model Level Logs to new Framework (#1783) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests `[Feature][Logger Framework]` Add convenience methods for logging strings (#1855) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests * Adds convenience methods to logger manager to better conform to python's `logging.Logger` [Feature] Info logs to console and debug logs to file (#1861) * Empty commit * - Add loggable_items in ModifiableModel Contract - Add pytorch implementation od loggable_items - Attach logger to state - Make changes in lifecycle convenience functions to include loggers - Add log_model_info method in Modifier which is called on each `update_event` but logs only on epoch end * - Move log call at the end of update_event - Remove extra space * - Move model level logging responsibility to `ModelLoggingMixin_` * - Style * - Update docstring * - Move mixin to it's own file * - Add test * - update logic to check epoch end * - Add more tests * - log model level logs at session level - filter logs to only include percentages - filter logs to only include params with non zero sparsity - filter logs to include only quantized modules - LoggerMixin functions to helpers.py * - Expose model log cadence - Propagate cadence via session to state and loggers - log model info in session - update log condition in logger.py * - Migrate tests * Fix epoch number logging * - fix failing test * Address review comments! * Remove event_type from should log model info * Update docstring * Style * mock _log_model_info * Move to using log scalar over log string Do not log at last_log_epoch but only if current epoch has reached specified cadence Remove model log cadence Style Update tests * Adds convenience methods to logger manager to better conform to python's `logging.Logger` * Move FileHandler creation above StreamHandler creation * Remove missed comment [Move] logger into it's own package (#1924) improves structure and separation of concerns Add local logs directory to .gitignore (#1925) [Add] frequency_manager to check log "readiness" (#1927) * Add frequency_manager to check log "readiness" * Refactor condition to be more readable [Logger Framework][Add] mode and frequency type to FrequencyManager (#1930) * Add mode and frequency type to FrequencyManager * move tests cases near the test * Style * [Use] frequency manager in logger (#1931) * Make log_frequency a property of FrequencyManager Add FrequencyManager to LoggerManager Add log_wriiten and model_updated to LoggerManager * [Set] [OLD MODIFIERS] _last_log_epoch to None * Update old modifiers to use named arguments to log_ready * [Update][Rename] * `PossibleLoggingMode` --> `LoggingModeType` * `PossibleFrequencyType` --> `FrequencyType` * Fix log format when step is None (#1933) * [Add] Wrap System and Metric Logging into it's own classes (#1932) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * [Logger Refactor] Session based Logs (#1920) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * Add session level logs * [Logger Framework] Log Losses (#1934) * Add __repr__ Move string logging functions to SystemLoggingWrapper Move metric logging functions to MetricLoggingWrapper * minor fixes * Remove warnings * Add back old logger * Add logs for loss * Add test * Use Sparsification Group logger (#1936) * Save last log step Add a stateless log_ready method * log only when loss is not None * Add tb style logging functions (#1919) * Add wandb log function (#1918) * [Test Update] Fix failing tests after adding back old logger (#1963) * [Logger Refactor] Add timer in logger manager (#1967) * Add timer in logger manager * Address review comments * update docstring * fix argument
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support to log losses in the new sparsification framework;
Based off of #1920
Example log file has been attached for GMP with
mobilenet_v2latest-log-gmp-mobilenet-12-01-2024_07.58.30.log