tracing: decide if each mutation should trigger a new span event to deal with the case of excessive mutations and storage

[odeke-em]

In our tracing update reviews, everyone initially raised the need to record a span event whenever a new mutation had been added and I spun up some pull requests. However, @harshachinta wisely noted that if a customer creates 40,000 mutations to be added before invoking .commit(), 40,000 separate events would be created hence this would constitute a horrible user experience and also lots of data that a customer could be charged for spans per https://github.com/googleapis/python-spanner/pull/1259/files#r1875359721

Facts

• by default, 128 span events is the maximum per span: it can be controlled by environment configuration with OpenTelemetry
• this isn't a show stopper and can be thought about and added so much later on once there is consensus
• there is recording of num_mutations when invoking .commit() that I had added as a span attribute per

  python-spanner/google/cloud/spanner_v1/batch.py

  Line 189 in a6811af

  trace_attributes = {"num_mutations": len(self._mutations)}

This issue is left as a reference for future decisions and to gain consensus with clear thought in time.