predicted ROI comes as 0 for all channels

[isaacmalka]

Hi, we want to switch from LightWeightMMM to Meridian soon and would like to ensure we get similar results like we did with LWMMM. Is there a way to use "default LWMMM" settings to make sure we get values for ROI, and MROI? It seems like we need to calibrate the ROI period, but I'm worried this might not scale if we do this per spend channel. Would appreciate any insights and pointers as for the same data, we're getting different outputs from LWMMM (non-zero of course). Also, we don't have the impressions data - so we just use cost, does the model assume number of impressions anyway?

[iwawithin]

media_summary_for_20_lastmon.csv
I have attached a summary table of all channels. I am encountering same issue where I am still receiving impressions from the analyzer for channels that should have zero impressions according to the summary table

[dattatreyam23]

Hello @isaacmalka,

Thank you for contacting us!

I would first like to share our documentation on Migrating From Lightweight MMM as this gives an overview of the differences between LMMM and Meridian, both in terms of modelling methodology and expected results. Please go through the above linked page if you haven’t done so already.

There are no “default LMMM” settings that can be directly used in Meridian to get the exact same results as obtained in LMMM. This is because Meridian models data differently, it has a different set of assumptions, is more complex and has more parameters to be estimated. While there is no one-stop solution to replicating LMMM results, we can help you debug the problems you are facing with Meridian.

The media input data expected is typically a non-negative exposure metric (such as impressions) of each media channel. The spend data is input as media_spend. Please note that in the absence of Reach and Frequency data both media and media_spend arguments are compulsory for Meridian, so you may choose to pass the same media spend data for the media argument in the absence of exposure metrics in which case you will see impressions derived from the media variable in your final summary table.

You may refer to our documentation on Collecting and Organizing your Data for more information on the expected input format in Meridian. Also sharing the API documentation page on InputData Object in Meridian for your reference.
To address your query regarding the ROI prior calibration, please note that LMMM uses beta coefficient priors and not ROI priors. Hence you may see different results in Meridian when you use ROI parameterization. You may also use beta priors in Meridian by setting paid_media_prior_type = 'coefficient' in your model specifications (Ref - ModelSpec). Our default approach, however, is to use ROI priors. Refer to our documentation on ROI Priors and Calibration to better understand why and how to use ROI parameterization in Meridian.

The 0 estimates you are getting for ROI and mROI seem erroneous and I am unable to identify the exact reason for this issue as I am getting non-zero estimates with sample data available on GitHub. Please attempt incorporating the above suggestions and if the incorrect ROI and mROI estimates still persists then please share more information about your choice of priors (whether you are using beta coefficient or ROI priors) and other relevant details on customizations you are specifying while defining the model so that we can better assist you.

Feel free to respond in this thread if you have any further queries regarding LMMM to Meridian migration.

Thank you,
Google Meridian Support Team

[isaacmalka]

Hi @dattatreyam23 - thank you for your reply.

I'd be happy if you could look into @iwawithin 's reply and let us know if we are doing something wrong. If we calibrate using recent 8 weeks worth of data (we're experimenting with the correct number), we are again receiving contributions higher than a 100% even though we were able to cap that by using the following updated code:
``
p_mean = 0.5 # 50% contribution mean
p_sd = 0.15 # 15% standard deviation # Define ROI priors
kpi = np.sum(input_data_xr["kpi"].values) # Sum KPI across all time periods
cost = np.sum(input_data_xr["media_spend"].values) # Sum of all media spend

        # Ensure KPI and cost are in float32 for compatibility with TensorFlow Probability
        kpi = tf.convert_to_tensor(kpi, dtype=tf.float32)
        cost = tf.convert_to_tensor(cost, dtype=tf.float32)

        # Calculate ROI mean and standard deviation
        roi_mean = tf.convert_to_tensor(p_mean * kpi / cost, dtype=tf.float32)
        roi_sd = tf.convert_to_tensor(
            p_sd * kpi / tf.sqrt(tf.reduce_sum(tf.square(cost))), dtype=tf.float32
        )

        # Calculate parameters for the LogNormal distribution
        lognormal_sigma = tf.sqrt(
            tf.math.log(tf.square(roi_sd) / tf.square(roi_mean) + 1)
        )
        lognormal_mu = tf.math.log(
            roi_mean * tf.exp(-tf.square(lognormal_sigma) / 2)
        )

        # Set the ROI prior based on total media contribution
        roi_prior = tfp.distributions.LogNormal(lognormal_mu, lognormal_sigma)
        tau_g_prior = tfp.distributions.Normal(
            0.0, 5.0, name=constants.TAU_G_EXCL_BASELINE
        )

        prior = prior_distribution.PriorDistribution(
            tau_g_excl_baseline=tau_g_prior, roi_m=roi_prior
        )
        np.random.seed(1)
        test_pct = 0.2  # changed test set to 20%
        n_times = len(input_data.time)

        holdout_id = np.full(n_times, False)
        holdout_times = np.random.choice(
            n_times, int(test_pct * n_times), replace=False
        )
        holdout_id[holdout_times] = True
        all_dates = pd.to_datetime(input_data_xr["time"].values)
        temp_last_date = all_dates.max()
        roi_period_dates = pd.date_range(
            end=temp_last_date, periods=8, freq="W-MON"
        )
        roi_calibration_period = np.zeros(
            (len(input_data_xr.time), len(input_data_xr.media_channel))
        )
        time_indices = np.where(
            np.isin(
                input_data_xr.time.values, roi_period_dates.strftime("%Y-%m-%d")
            )
        )[0]
        roi_calibration_period[time_indices, :] = 1
        roi_calibration_period[:, :] = np.where(
            roi_calibration_period.sum(axis=0) == 0, 1, roi_calibration_period
        )

        # Create model specification
        model_spec = spec.ModelSpec(
            knots=52,  # 52
            holdout_id=holdout_id,
            prior=prior,
            rf_roi_calibration_period=None,  # Ensure no reach/frequency calibration
            roi_calibration_period=roi_calibration_period,
            use_roi_prior=True,  # disabling ROI priors
        )

``
Also, how is impression mapped into a column with values if we don't have this data?

Many thanks for your help and reply.

[cpulavarthi]

Hello @isaacmalka,

Thank you for sharing additional context and the code you are using.

Your code for ROI prior setting and calibration appears to be correct and works as expected with the sample simulated dataset shared in the Meridian GitHub Repository (national_media.csv dataset).

However, please note that you appear to be computing an ROI prior based on total media contribution (using KPI and cost summed over all periods) as linked here but calibrating the derived ROI to just the last 8 periods. Please ensure that these two windows are aligned.

Also your code for computing cost appears to return a scalar. However cost is expected to be the total cost per channel array (Ref - Total Paid Media Media Contribution Prior). Please be sure to correct the aggregation accordingly for your dataset.

1. With your code I am not encountering the issue of 0 ROI and mROI estimates mentioned in your initial query, please let me know if that issue still persists.

2. Please check our documentation on Low or Negative Baseline Attribution for some debugging steps when you encounter over 100% incremental contribution for your media channels.

3. In the absence of reach and frequency data, both media and media_spend arguments are mandatory in Meridian for media channels. You must be using media spend data for both media and media_spend arguments. This is why the final summary table has non-zero impressions values as they are getting computed from the media data used.

4. I can see from our discussion that you are using national level weekly data and modeling it with 52 knots. How many years of data do you have and how are you choosing what periods to model with knots? There can be convergence problems if there is insufficient within-week variation in the data (for example, if each week appears just once in the dataset). Please see guidance on knots here.

Feel free to reach out if you have any further questions regarding this.

Thank you,

Google Meridian Support Team

[iwawithin]

we are using ROI priors on our data as following

`
p_mean = 0.5
p_sd = 0.15
kpi = np.sum(input_data_xr["kpi"].values)
cost = np.sum(input_data_xr["media_spend"].values)

    kpi = tf.convert_to_tensor(kpi, dtype=tf.float32)
    cost = tf.convert_to_tensor(cost, dtype=tf.float32)

    roi_mean = tf.convert_to_tensor(p_mean * kpi / cost, dtype=tf.float32)
    roi_sd = tf.convert_to_tensor(
        p_sd * kpi / tf.sqrt(tf.reduce_sum(tf.square(cost))), dtype=tf.float32
    )

    lognormal_sigma = tf.sqrt(
        tf.math.log(tf.square(roi_sd) / tf.square(roi_mean) + 1)
    )
    lognormal_mu = tf.math.log(
        roi_mean * tf.exp(-tf.square(lognormal_sigma) / 2)
    )

    roi_prior = tfp.distributions.LogNormal(lognormal_mu, lognormal_sigma)
    tau_g_prior = tfp.distributions.Normal(
        0.0, 5.0, name=constants.TAU_G_EXCL_BASELINE
    )

    prior = prior_distribution.PriorDistribution(
        tau_g_excl_baseline=tau_g_prior, roi_m=roi_prior
    )

`