[Add Feature] - Throw an error if softmax is used with 1 neuron

[Frightera]

This is a utility function to check if the usage of softmax makes sense (new users make this mistake a lot). Applying softmax on a single neuron will make the model output ones everytime, there are too many Stackoverflow posts about this.

In order to see this in action, please check the gist.

This applies for any other layers (Conv2D etc.) where the applied axis (axis=-1 default) of softmax has only one unit.

[rchao]

Thanks for the PR! @Frightera can you confirm that the newly added test would fail without the fix in the library changes?

@pradeepkuppla, would it be possible to give this a run across all internal tests before we proceed with an approval?

[Frightera]

Hi @rchao I am not sure if I understood you correctly - do you want me to write a test to fail it on purpose?

[sachinprasadhs]

@Frightera ,

It is failing internal test with below error, could you please make the necessary changes.

    "softmax" in str(output.name.lower())
AttributeError: 'NoneType' object has no attribute 'lower'

[Frightera]

@sachinprasadhs Should be fine now, I re-run the tests manually.

That was an edge case I didn't think of 😮

Edit: I might try to simplify nested if statements but that seemed the cleanest way to me.

[rchao]

Hi @rchao I am not sure if I understood you correctly - do you want me to write a test to fail it on purpose?

To make sure we properly cover the fix, the idea is to have a unit test that would have failed without your library changes. Can you confirm this is the case here?

[Frightera]

@rchao To make it clear - This is not actually a fix, there was no error in Keras. That's just a warning to catch a logical error made by new users.

So you really can not catch this with a regular test as the model will be compiled successfully.

[Frightera]

@sachinprasadhs That's interesting, this error was not present in my local CI. Let's try again.

[sachinprasadhs]

@Frightera , Could you please check the below error.

Traceback (most recent call last):
  File "/build/work/45a231adaff5e6784bf9c607c668de34477f/google3/runfiles/google3/third_party/py/keras/engine/training_test.py", [line 5110](https://cs.corp.google.com/piper///depot/google3/third_party/py/keras/engine/training_test.py?l=5110&ws=tap-presubmit-server/67803804&snapshot=2), in test_functional_model_output
    with self.assertRaisesRegex(
AssertionError: ValueError not raised

[Frightera]

Hi @sachinprasadhs,

I don't know why this is happening for the second time, I don't get any errors in Colab and my local environment when I run the CI cycle.

Also the PR #18264 has now finished its build without any errors. Why is there a contradiction like this?

[sachinprasadhs]

Now the previous error is not showing anymore.

The internal test result for the above linked PR by copybara is what I'm providing.

Here is the new result output.

File "/build/work/918d6c4f2fd469de06938bb711d72b99a84f/google3/runfiles/google3/third_party/py/keras/engine/training.py", [line 4523](https://cs.corp.google.com/piper///depot/google3/third_party/py/keras/engine/training.py?l=4523&ws=tap-presubmit-server/67837954&snapshot=2), in _check_output_activation_softmax
    raise ValueError(
ValueError: Output layer dense_2 has a single unit output, but the activation is softmax. This is most likely an error because softmax outputs sum to 1 therefore single unit outputs with softmax will only output 1.0. If you think that the error is raised due to an incorrect check, please file an issue on [https://github.com/keras-team/keras/issues](https://www.google.com/url?q=https://github.com/keras-team/keras/issues&sa=D). You can disable this check by setting `experimental_validate_softmax_activation=False` when calling `compile()` on the model.

[Frightera]

@sachinprasadhs Thanks for the quick response.

Maybe I am missing something but PR opened by copybara https://github.com/keras-team/keras/pull/18264 #18264 does not include any errors in the internal results.

Also the error you sent:

File "/build/work/918d6c4f2fd469de06938bb711d72b99a84f/google3/runfiles/google3/third_party/py/keras/engine/training.py", [line 4523](https://cs.corp.google.com/piper///depot/google3/third_party/py/keras/engine/training.py?l=4523&ws=tap-presubmit-server/67837954&snapshot=2), in _check_output_activation_softmax
    raise ValueError(
ValueError: Output layer dense_2 has a single unit output, but the activation is softmax. This is most likely an error because softmax outputs sum to 1 therefore single unit outputs with softmax will only output 1.0. If you think that the error is raised due to an incorrect check, please file an issue on [https://github.com/keras-team/keras/issues](https://www.google.com/url?q=https://github.com/keras-team/keras/issues&sa=D). You can disable this check by setting `experimental_validate_softmax_activation=False` when calling `compile()` on the model.

This part does not exist in my PR: 🤔

please file an issue on [https://github.com/keras-team/keras/issues](https://www.google.com/url?q=https://github.com/keras-team/keras/issues&sa=D).

This is slightly different from what I have in my PR:

f"Output layer {layer_name} has a single unit output, "
                    "but the activation is softmax. This is most likely an "
                    "error because softmax outputs sum to 1 therefore single "
                    "unit outputs with softmax will only output 1.0. If you "
                    "think that the error is raised due to an incorrect check, "
                    "please file an issue on "
                    "https://github.com/keras-team/keras/issues. You can "
                    "disable this check by setting "
                    "`experimental_validate_softmax_activation=False` when "
                    "calling `compile()` on the model."
                )

So the problem is I can't see the error in the internal tests, and the provided value error is slightly different from what I have in my PR.

[sachinprasadhs]

The internal Tests will be run against different hardware configurations, different OS environments, this error seems to be happening in //third_party/py/keras/integration_test:dragonfish_preprocessing_applied_in_dataset_creator_test_tpu

[Frightera]

So after tracing this I found:

keras/keras/integration_test/preprocessing_test_utils.py

Lines 88 to 113 in 4829ddf

	def make_training_model():
	"""Make a trainable model for the preprocessed inputs."""
	float_in = tf.keras.Input(shape=(1,), dtype="float32", name="float_col")
	# After preprocessing, both the string and int column are integer ready for
	# embedding.
	int_in = tf.keras.Input(shape=(1,), dtype="int64", name="int_col")
	string_in = tf.keras.Input(shape=(1,), dtype="int64", name="string_col")
	# Feed the lookup layers into an embedding.
	int_embedding = tf.keras.layers.Embedding(VOCAB_SIZE + 1, 8, input_length=1)
	int_out = int_embedding(int_in)
	int_out = tf.keras.layers.Flatten()(int_out)
	string_embedding = tf.keras.layers.Embedding(
	VOCAB_SIZE + 1, 8, input_length=1
	)
	string_out = string_embedding(string_in)
	string_out = tf.keras.layers.Flatten()(string_out)
	# Concatenate outputs.
	concatate = tf.keras.layers.Concatenate()
	# Feed our preprocessed inputs into a simple MLP.
	x = concatate((float_in, int_out, string_out))
	x = tf.keras.layers.Dense(32, activation="relu")(x)
	x = tf.keras.layers.Dense(32, activation="relu")(x)
	outputs = tf.keras.layers.Dense(1, activation="softmax")(x)
	return tf.keras.Model(inputs=(float_in, int_in, string_in), outputs=outputs)

In line 112, we can see: outputs = tf.keras.layers.Dense(1, activation="softmax")(x). So this was triggering my check in the PR and was raising an error. Obviously this is a wrong model setup in the internal test, I can fix the setup in this PR or can open a seperate PR for it.

Which action should we take in this case?

Edit: make_training_model() is called in preprocessing_applied_in_dataset_creator_test

[sachinprasadhs]

@rchao , Can you please suggest in the above comment.

[Frightera]

Hi @rchao, any update on this? A different internal test needs to be changed.

[Frightera]

Hi, any update on this PR by any chance?

[sachinprasadhs]

@qlzh727 , Could you please take a look into this PR. Thanks!

[sachinprasadhs]

Hello, Thank you for submitting a pull request.

We're currently in the process of migrating the new Keras 3 code base from keras-team/keras-core to keras-team/keras.
Consequently, merging this PR is not possible at the moment. After the migration is successfully completed, feel free to reopen this PR at keras-team/keras if you believe it remains relevant to the Keras 3 code base. If instead this PR fixes a bug or security issue in legacy tf.keras, you can instead reopen the PR at keras-team/tf-keras, which hosts the TensorFlow-only, legacy version of Keras.