California Housing

Author: givasile
Runtime: ~35 s
Description: Global and regional RHALE effects on the California-housing dataset, explaining a neural network's predicted house values — with regional splits on income, latitude and longitude.
📄 The whole notebook in one page: RHALE report

import numpy as np
import keras
import tensorflow as tf
import effector
from sklearn.datasets import fetch_california_housing

california_housing = fetch_california_housing(as_frame=True)

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1784066550.180750   22311 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

np.random.seed(21)

print(california_housing.DESCR)

.. _california_housing_dataset:

California Housing dataset
--------------------------

**Data Set Characteristics:**

:Number of Instances: 20640

:Number of Attributes: 8 numeric, predictive attributes and the target

:Attribute Information:
    - MedInc        median income in block group
    - HouseAge      median house age in block group
    - AveRooms      average number of rooms per household
    - AveBedrms     average number of bedrooms per household
    - Population    block group population
    - AveOccup      average number of household members
    - Latitude      block group latitude
    - Longitude     block group longitude

:Missing Attribute Values: None

This dataset was obtained from the StatLib repository.
https://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.html

The target variable is the median house value for California districts,
expressed in hundreds of thousands of dollars ($100,000).

This dataset was derived from the 1990 U.S. census, using one row per census
block group. A block group is the smallest geographical unit for which the U.S.
Census Bureau publishes sample data (a block group typically has a population
of 600 to 3,000 people).

A household is a group of people residing within a home. Since the average
number of rooms and bedrooms in this dataset are provided per household, these
columns may take surprisingly large values for block groups with few households
and many empty houses, such as vacation resorts.

It can be downloaded/loaded using the
:func:`sklearn.datasets.fetch_california_housing` function.

.. rubric:: References

- Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,
  Statistics and Probability Letters, 33:291-297, 1997.

feature_names = california_housing.feature_names
target_name= california_housing.target_names[0]
df = type(california_housing.frame)

X = california_housing.data
y = california_housing.target

print("Design matrix shape: {}".format(X.shape))
print("---------------------------------")
for col_name in X.columns:
    print("Feature: {:15}, unique: {:4d}, Mean: {:6.2f}, Std: {:6.2f}, Min: {:6.2f}, Max: {:6.2f}".format(col_name, len(X[col_name].unique()), X[col_name].mean(), X[col_name].std(), X[col_name].min(), X[col_name].max()))

print("\nTarget shape: {}".format(y.shape))
print("---------------------------------")
print("Target: {:15}, unique: {:4d}, Mean: {:6.2f}, Std: {:6.2f}, Min: {:6.2f}, Max: {:6.2f}".format(y.name, len(y.unique()), y.mean(), y.std(), y.min(), y.max()))

Design matrix shape: (20640, 8)
---------------------------------
Feature: MedInc         , unique: 12928, Mean:   3.87, Std:   1.90, Min:   0.50, Max:  15.00
Feature: HouseAge       , unique:   52, Mean:  28.64, Std:  12.59, Min:   1.00, Max:  52.00
Feature: AveRooms       , unique: 19392, Mean:   5.43, Std:   2.47, Min:   0.85, Max: 141.91
Feature: AveBedrms      , unique: 14233, Mean:   1.10, Std:   0.47, Min:   0.33, Max:  34.07
Feature: Population     , unique: 3888, Mean: 1425.48, Std: 1132.46, Min:   3.00, Max: 35682.00
Feature: AveOccup       , unique: 18841, Mean:   3.07, Std:  10.39, Min:   0.69, Max: 1243.33
Feature: Latitude       , unique:  862, Mean:  35.63, Std:   2.14, Min:  32.54, Max:  41.95
Feature: Longitude      , unique:  844, Mean: -119.57, Std:   2.00, Min: -124.35, Max: -114.31

Target shape: (20640,)
---------------------------------
Target: MedHouseVal    , unique: 3842, Mean:   2.07, Std:   1.15, Min:   0.15, Max:   5.00

def preprocess(X, y):
    # Compute mean and std for outlier detection
    X_mean = X.mean()
    X_std = X.std()

    # Exclude instances with any feature 2 std away from the mean
    mask = (X - X_mean).abs() <= 2 * X_std
    mask = mask.all(axis=1)

    X_filtered = X[mask]
    y_filtered = y[mask]

    # Standardize X
    X_mean = X_filtered.mean()
    X_std = X_filtered.std()
    X_standardized = (X_filtered - X_mean) / X_std

    # Standardize y
    y_mean = y_filtered.mean()
    y_std = y_filtered.std()
    y_standardized = (y_filtered - y_mean) / y_std

    return X_standardized, y_standardized, X_mean, X_std, y_mean, y_std



# shuffle and standarize all features
X_df, Y_df, x_mean, x_std, y_mean, y_std = preprocess(X, y)

def split(X_df, Y_df):
    # shuffle indices
    indices = np.arange(len(X_df))
    np.random.shuffle(indices)

    # data split
    train_size = int(0.8 * len(X_df))

    X_train = X_df.iloc[indices[:train_size]]
    Y_train = Y_df.iloc[indices[:train_size]]
    X_test = X_df.iloc[indices[train_size:]]
    Y_test = Y_df.iloc[indices[train_size:]]

    return X_train, Y_train, X_test, Y_test

# train/test split
X_train, Y_train, X_test, Y_test = split(X_df, Y_df)

# Train - Evaluate - Explain a neural network
model = keras.Sequential([
    keras.layers.Dense(1024, activation="relu"),
    keras.layers.Dense(512, activation="relu"),
    keras.layers.Dense(256, activation="relu"),
    keras.layers.Dense(1)
])

optimizer = keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=optimizer, loss="mse", metrics=["mae", keras.metrics.RootMeanSquaredError()])
model.fit(X_train, Y_train, batch_size=1024, epochs=20, verbose=1)
model.evaluate(X_train, Y_train, verbose=1)
model.evaluate(X_test, Y_test, verbose=1)

Epoch 1/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m13s[0m 979ms/step - loss: 0.9679 - mae: 0.7580 - root_mean_squared_error: 0.9838

[1m 4/15[0m [32m━━━━━[0m[37m━━━━━━━━━━━━━━━[0m [1m0s[0m 22ms/step - loss: 0.8043 - mae: 0.6834 - root_mean_squared_error: 0.8948

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 21ms/step - loss: 0.7198 - mae: 0.6399 - root_mean_squared_error: 0.8452

[1m10/15[0m [32m━━━━━━━━━━━━━[0m[37m━━━━━━━[0m [1m0s[0m 21ms/step - loss: 0.6626 - mae: 0.6097 - root_mean_squared_error: 0.8098

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 21ms/step - loss: 0.6211 - mae: 0.5868 - root_mean_squared_error: 0.7833

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m1s[0m 22ms/step - loss: 0.4590 - mae: 0.4957 - root_mean_squared_error: 0.6775

Epoch 2/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 42ms/step - loss: 0.3745 - mae: 0.4383 - root_mean_squared_error: 0.6120

[1m 4/15[0m [32m━━━━━[0m[37m━━━━━━━━━━━━━━━[0m [1m0s[0m 22ms/step - loss: 0.3426 - mae: 0.4206 - root_mean_squared_error: 0.5851

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 22ms/step - loss: 0.3366 - mae: 0.4184 - root_mean_squared_error: 0.5800

[1m10/15[0m [32m━━━━━━━━━━━━━[0m[37m━━━━━━━[0m [1m0s[0m 23ms/step - loss: 0.3341 - mae: 0.4171 - root_mean_squared_error: 0.5779

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 23ms/step - loss: 0.3310 - mae: 0.4149 - root_mean_squared_error: 0.5752

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 24ms/step - loss: 0.3183 - mae: 0.4051 - root_mean_squared_error: 0.5642

Epoch 3/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 51ms/step - loss: 0.2825 - mae: 0.3806 - root_mean_squared_error: 0.5315

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2885 - mae: 0.3859 - root_mean_squared_error: 0.5371

[1m 6/15[0m [32m━━━━━━━━[0m[37m━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2940 - mae: 0.3871 - root_mean_squared_error: 0.5422

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2939 - mae: 0.3865 - root_mean_squared_error: 0.5421

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2941 - mae: 0.3862 - root_mean_squared_error: 0.5423

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 26ms/step - loss: 0.2943 - mae: 0.3863 - root_mean_squared_error: 0.5425

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 27ms/step - loss: 0.2951 - mae: 0.3867 - root_mean_squared_error: 0.5432

Epoch 4/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 49ms/step - loss: 0.3160 - mae: 0.3945 - root_mean_squared_error: 0.5621

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.3053 - mae: 0.3914 - root_mean_squared_error: 0.5525

[1m 6/15[0m [32m━━━━━━━━[0m[37m━━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.3006 - mae: 0.3877 - root_mean_squared_error: 0.5483

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2983 - mae: 0.3867 - root_mean_squared_error: 0.5462

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2965 - mae: 0.3855 - root_mean_squared_error: 0.5445

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 25ms/step - loss: 0.2953 - mae: 0.3848 - root_mean_squared_error: 0.5433

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 26ms/step - loss: 0.2868 - mae: 0.3806 - root_mean_squared_error: 0.5355

Epoch 5/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 53ms/step - loss: 0.3014 - mae: 0.3724 - root_mean_squared_error: 0.5490

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2913 - mae: 0.3748 - root_mean_squared_error: 0.5397

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2890 - mae: 0.3756 - root_mean_squared_error: 0.5375

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2859 - mae: 0.3747 - root_mean_squared_error: 0.5346

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2835 - mae: 0.3740 - root_mean_squared_error: 0.5324

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2825 - mae: 0.3738 - root_mean_squared_error: 0.5314

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 27ms/step - loss: 0.2818 - mae: 0.3736 - root_mean_squared_error: 0.5308

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 27ms/step - loss: 0.2740 - mae: 0.3704 - root_mean_squared_error: 0.5234

Epoch 6/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 50ms/step - loss: 0.2568 - mae: 0.3522 - root_mean_squared_error: 0.5068

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2626 - mae: 0.3566 - root_mean_squared_error: 0.5124

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2663 - mae: 0.3611 - root_mean_squared_error: 0.5160

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2673 - mae: 0.3630 - root_mean_squared_error: 0.5170

[1m10/15[0m [32m━━━━━━━━━━━━━[0m[37m━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2684 - mae: 0.3644 - root_mean_squared_error: 0.5181

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 25ms/step - loss: 0.2690 - mae: 0.3651 - root_mean_squared_error: 0.5186

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 26ms/step - loss: 0.2712 - mae: 0.3670 - root_mean_squared_error: 0.5207

Epoch 7/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 52ms/step - loss: 0.2608 - mae: 0.3684 - root_mean_squared_error: 0.5106

[1m 4/15[0m [32m━━━━━[0m[37m━━━━━━━━━━━━━━━[0m [1m0s[0m 24ms/step - loss: 0.2686 - mae: 0.3669 - root_mean_squared_error: 0.5182

[1m 6/15[0m [32m━━━━━━━━[0m[37m━━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2665 - mae: 0.3647 - root_mean_squared_error: 0.5162

[1m 8/15[0m [32m━━━━━━━━━━[0m[37m━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2655 - mae: 0.3635 - root_mean_squared_error: 0.5152

[1m10/15[0m [32m━━━━━━━━━━━━━[0m[37m━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2657 - mae: 0.3635 - root_mean_squared_error: 0.5155

[1m12/15[0m [32m━━━━━━━━━━━━━━━━[0m[37m━━━━[0m [1m0s[0m 25ms/step - loss: 0.2662 - mae: 0.3636 - root_mean_squared_error: 0.5159

[1m14/15[0m [32m━━━━━━━━━━━━━━━━━━[0m[37m━━[0m [1m0s[0m 25ms/step - loss: 0.2662 - mae: 0.3636 - root_mean_squared_error: 0.5159

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 26ms/step - loss: 0.2654 - mae: 0.3629 - root_mean_squared_error: 0.5151

Epoch 8/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 57ms/step - loss: 0.2644 - mae: 0.3626 - root_mean_squared_error: 0.5142

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 28ms/step - loss: 0.2566 - mae: 0.3591 - root_mean_squared_error: 0.5065

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2585 - mae: 0.3603 - root_mean_squared_error: 0.5084

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2610 - mae: 0.3613 - root_mean_squared_error: 0.5108

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2621 - mae: 0.3619 - root_mean_squared_error: 0.5119

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2626 - mae: 0.3621 - root_mean_squared_error: 0.5124

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 26ms/step - loss: 0.2629 - mae: 0.3621 - root_mean_squared_error: 0.5127

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 27ms/step - loss: 0.2632 - mae: 0.3609 - root_mean_squared_error: 0.5130

Epoch 9/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 50ms/step - loss: 0.3095 - mae: 0.4304 - root_mean_squared_error: 0.5563

[1m 4/15[0m [32m━━━━━[0m[37m━━━━━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2880 - mae: 0.4024 - root_mean_squared_error: 0.5365

[1m 6/15[0m [32m━━━━━━━━[0m[37m━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2841 - mae: 0.3934 - root_mean_squared_error: 0.5329

[1m 8/15[0m [32m━━━━━━━━━━[0m[37m━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2813 - mae: 0.3892 - root_mean_squared_error: 0.5303

[1m10/15[0m [32m━━━━━━━━━━━━━[0m[37m━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2785 - mae: 0.3856 - root_mean_squared_error: 0.5276

[1m12/15[0m [32m━━━━━━━━━━━━━━━━[0m[37m━━━━[0m [1m0s[0m 27ms/step - loss: 0.2769 - mae: 0.3830 - root_mean_squared_error: 0.5261

[1m14/15[0m [32m━━━━━━━━━━━━━━━━━━[0m[37m━━[0m [1m0s[0m 27ms/step - loss: 0.2757 - mae: 0.3810 - root_mean_squared_error: 0.5250

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 28ms/step - loss: 0.2672 - mae: 0.3674 - root_mean_squared_error: 0.5169

Epoch 10/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 55ms/step - loss: 0.2379 - mae: 0.3387 - root_mean_squared_error: 0.4878

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2428 - mae: 0.3432 - root_mean_squared_error: 0.4927

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2466 - mae: 0.3467 - root_mean_squared_error: 0.4966

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 28ms/step - loss: 0.2462 - mae: 0.3477 - root_mean_squared_error: 0.4962

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2461 - mae: 0.3480 - root_mean_squared_error: 0.4961

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2467 - mae: 0.3489 - root_mean_squared_error: 0.4966

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 29ms/step - loss: 0.2480 - mae: 0.3500 - root_mean_squared_error: 0.4980

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 29ms/step - loss: 0.2571 - mae: 0.3563 - root_mean_squared_error: 0.5070

Epoch 11/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 59ms/step - loss: 0.2427 - mae: 0.3458 - root_mean_squared_error: 0.4926

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2423 - mae: 0.3470 - root_mean_squared_error: 0.4922

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2438 - mae: 0.3473 - root_mean_squared_error: 0.4937

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2464 - mae: 0.3487 - root_mean_squared_error: 0.4963

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2478 - mae: 0.3497 - root_mean_squared_error: 0.4978

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2484 - mae: 0.3498 - root_mean_squared_error: 0.4984

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 26ms/step - loss: 0.2483 - mae: 0.3495 - root_mean_squared_error: 0.4982

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 27ms/step - loss: 0.2458 - mae: 0.3467 - root_mean_squared_error: 0.4957

Epoch 12/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 51ms/step - loss: 0.2346 - mae: 0.3374 - root_mean_squared_error: 0.4843

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2433 - mae: 0.3419 - root_mean_squared_error: 0.4932

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2455 - mae: 0.3452 - root_mean_squared_error: 0.4955

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2458 - mae: 0.3454 - root_mean_squared_error: 0.4957

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2455 - mae: 0.3453 - root_mean_squared_error: 0.4954

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2450 - mae: 0.3450 - root_mean_squared_error: 0.4950

[1m14/15[0m [32m━━━━━━━━━━━━━━━━━━[0m[37m━━[0m [1m0s[0m 26ms/step - loss: 0.2442 - mae: 0.3444 - root_mean_squared_error: 0.4942

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 27ms/step - loss: 0.2414 - mae: 0.3426 - root_mean_squared_error: 0.4913

Epoch 13/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 53ms/step - loss: 0.2214 - mae: 0.3443 - root_mean_squared_error: 0.4706

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2284 - mae: 0.3396 - root_mean_squared_error: 0.4779

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2318 - mae: 0.3392 - root_mean_squared_error: 0.4815

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2337 - mae: 0.3401 - root_mean_squared_error: 0.4833

[1m10/15[0m [32m━━━━━━━━━━━━━[0m[37m━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2347 - mae: 0.3399 - root_mean_squared_error: 0.4844

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 25ms/step - loss: 0.2346 - mae: 0.3396 - root_mean_squared_error: 0.4843

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 26ms/step - loss: 0.2361 - mae: 0.3382 - root_mean_squared_error: 0.4859

Epoch 14/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 52ms/step - loss: 0.2242 - mae: 0.3530 - root_mean_squared_error: 0.4735

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2294 - mae: 0.3466 - root_mean_squared_error: 0.4790

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 28ms/step - loss: 0.2304 - mae: 0.3436 - root_mean_squared_error: 0.4799

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 28ms/step - loss: 0.2317 - mae: 0.3432 - root_mean_squared_error: 0.4813

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 28ms/step - loss: 0.2328 - mae: 0.3424 - root_mean_squared_error: 0.4825

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2330 - mae: 0.3416 - root_mean_squared_error: 0.4826

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 27ms/step - loss: 0.2331 - mae: 0.3411 - root_mean_squared_error: 0.4828

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 28ms/step - loss: 0.2335 - mae: 0.3376 - root_mean_squared_error: 0.4832

Epoch 15/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 57ms/step - loss: 0.2248 - mae: 0.3274 - root_mean_squared_error: 0.4741

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2357 - mae: 0.3329 - root_mean_squared_error: 0.4854

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2341 - mae: 0.3340 - root_mean_squared_error: 0.4838

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2330 - mae: 0.3338 - root_mean_squared_error: 0.4826

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2313 - mae: 0.3332 - root_mean_squared_error: 0.4809

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2303 - mae: 0.3331 - root_mean_squared_error: 0.4798

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 26ms/step - loss: 0.2300 - mae: 0.3332 - root_mean_squared_error: 0.4796

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 27ms/step - loss: 0.2334 - mae: 0.3374 - root_mean_squared_error: 0.4831

Epoch 16/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 50ms/step - loss: 0.1917 - mae: 0.2998 - root_mean_squared_error: 0.4378

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2029 - mae: 0.3071 - root_mean_squared_error: 0.4504

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2065 - mae: 0.3124 - root_mean_squared_error: 0.4543

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2077 - mae: 0.3146 - root_mean_squared_error: 0.4557

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2097 - mae: 0.3164 - root_mean_squared_error: 0.4579

[1m12/15[0m [32m━━━━━━━━━━━━━━━━[0m[37m━━━━[0m [1m0s[0m 25ms/step - loss: 0.2133 - mae: 0.3196 - root_mean_squared_error: 0.4618

[1m14/15[0m [32m━━━━━━━━━━━━━━━━━━[0m[37m━━[0m [1m0s[0m 25ms/step - loss: 0.2157 - mae: 0.3215 - root_mean_squared_error: 0.4643

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 26ms/step - loss: 0.2314 - mae: 0.3337 - root_mean_squared_error: 0.4811

Epoch 17/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 54ms/step - loss: 0.2253 - mae: 0.3374 - root_mean_squared_error: 0.4747

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2255 - mae: 0.3324 - root_mean_squared_error: 0.4749

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2258 - mae: 0.3317 - root_mean_squared_error: 0.4752

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 27ms/step - loss: 0.2262 - mae: 0.3326 - root_mean_squared_error: 0.4756

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 28ms/step - loss: 0.2262 - mae: 0.3325 - root_mean_squared_error: 0.4756

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2263 - mae: 0.3327 - root_mean_squared_error: 0.4757

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 30ms/step - loss: 0.2265 - mae: 0.3330 - root_mean_squared_error: 0.4759

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 29ms/step - loss: 0.2268 - mae: 0.3332 - root_mean_squared_error: 0.4763

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 31ms/step - loss: 0.2295 - mae: 0.3347 - root_mean_squared_error: 0.4790

Epoch 18/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 58ms/step - loss: 0.2279 - mae: 0.3352 - root_mean_squared_error: 0.4774

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 30ms/step - loss: 0.2305 - mae: 0.3318 - root_mean_squared_error: 0.4801

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2263 - mae: 0.3279 - root_mean_squared_error: 0.4757

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 30ms/step - loss: 0.2242 - mae: 0.3268 - root_mean_squared_error: 0.4734

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2231 - mae: 0.3264 - root_mean_squared_error: 0.4723

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 30ms/step - loss: 0.2224 - mae: 0.3258 - root_mean_squared_error: 0.4715

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 31ms/step - loss: 0.2218 - mae: 0.3256 - root_mean_squared_error: 0.4709

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 31ms/step - loss: 0.2206 - mae: 0.3273 - root_mean_squared_error: 0.4697

Epoch 19/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 62ms/step - loss: 0.2216 - mae: 0.3230 - root_mean_squared_error: 0.4707

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 31ms/step - loss: 0.2154 - mae: 0.3204 - root_mean_squared_error: 0.4640

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 30ms/step - loss: 0.2166 - mae: 0.3233 - root_mean_squared_error: 0.4654

[1m 7/15[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2174 - mae: 0.3242 - root_mean_squared_error: 0.4662

[1m 9/15[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2192 - mae: 0.3255 - root_mean_squared_error: 0.4682

[1m11/15[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 29ms/step - loss: 0.2200 - mae: 0.3262 - root_mean_squared_error: 0.4690

[1m13/15[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 29ms/step - loss: 0.2209 - mae: 0.3267 - root_mean_squared_error: 0.4699

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 29ms/step - loss: 0.2252 - mae: 0.3305 - root_mean_squared_error: 0.4745

Epoch 20/20

[1m 1/15[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 54ms/step - loss: 0.2143 - mae: 0.3162 - root_mean_squared_error: 0.4629

[1m 3/15[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2174 - mae: 0.3167 - root_mean_squared_error: 0.4663

[1m 5/15[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 26ms/step - loss: 0.2158 - mae: 0.3165 - root_mean_squared_error: 0.4645

[1m 8/15[0m [32m━━━━━━━━━━[0m[37m━━━━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2137 - mae: 0.3163 - root_mean_squared_error: 0.4623

[1m10/15[0m [32m━━━━━━━━━━━━━[0m[37m━━━━━━━[0m [1m0s[0m 25ms/step - loss: 0.2128 - mae: 0.3161 - root_mean_squared_error: 0.4613

[1m12/15[0m [32m━━━━━━━━━━━━━━━━[0m[37m━━━━[0m [1m0s[0m 25ms/step - loss: 0.2121 - mae: 0.3161 - root_mean_squared_error: 0.4605

[1m14/15[0m [32m━━━━━━━━━━━━━━━━━━[0m[37m━━[0m [1m0s[0m 25ms/step - loss: 0.2119 - mae: 0.3162 - root_mean_squared_error: 0.4603

[1m15/15[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 26ms/step - loss: 0.2112 - mae: 0.3175 - root_mean_squared_error: 0.4595

[1m 1/456[0m [37m━━━━━━━━━━━━━━━━━━━━[0m [1m1:18[0m 173ms/step - loss: 0.1588 - mae: 0.2646 - root_mean_squared_error: 0.3985

[1m 19/456[0m [37m━━━━━━━━━━━━━━━━━━━━[0m [1m1s[0m 3ms/step - loss: 0.1949 - mae: 0.3032 - root_mean_squared_error: 0.4410

[1m 37/456[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m1s[0m 3ms/step - loss: 0.2006 - mae: 0.3093 - root_mean_squared_error: 0.4476

[1m 55/456[0m [32m━━[0m[37m━━━━━━━━━━━━━━━━━━[0m [1m1s[0m 3ms/step - loss: 0.2037 - mae: 0.3119 - root_mean_squared_error: 0.4511

[1m 74/456[0m [32m━━━[0m[37m━━━━━━━━━━━━━━━━━[0m [1m1s[0m 3ms/step - loss: 0.2048 - mae: 0.3133 - root_mean_squared_error: 0.4524

[1m 91/456[0m [32m━━━[0m[37m━━━━━━━━━━━━━━━━━[0m [1m1s[0m 3ms/step - loss: 0.2058 - mae: 0.3146 - root_mean_squared_error: 0.4535

[1m110/456[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2068 - mae: 0.3159 - root_mean_squared_error: 0.4546

[1m131/456[0m [32m━━━━━[0m[37m━━━━━━━━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2066 - mae: 0.3165 - root_mean_squared_error: 0.4545

[1m152/456[0m [32m━━━━━━[0m[37m━━━━━━━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2070 - mae: 0.3171 - root_mean_squared_error: 0.4549

[1m172/456[0m [32m━━━━━━━[0m[37m━━━━━━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2071 - mae: 0.3174 - root_mean_squared_error: 0.4550

[1m192/456[0m [32m━━━━━━━━[0m[37m━━━━━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2071 - mae: 0.3176 - root_mean_squared_error: 0.4550

[1m214/456[0m [32m━━━━━━━━━[0m[37m━━━━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2072 - mae: 0.3179 - root_mean_squared_error: 0.4551

[1m235/456[0m [32m━━━━━━━━━━[0m[37m━━━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2074 - mae: 0.3180 - root_mean_squared_error: 0.4553

[1m256/456[0m [32m━━━━━━━━━━━[0m[37m━━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2073 - mae: 0.3181 - root_mean_squared_error: 0.4553

[1m278/456[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2071 - mae: 0.3181 - root_mean_squared_error: 0.4550

[1m301/456[0m [32m━━━━━━━━━━━━━[0m[37m━━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2069 - mae: 0.3181 - root_mean_squared_error: 0.4548

[1m324/456[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2067 - mae: 0.3181 - root_mean_squared_error: 0.4546

[1m348/456[0m [32m━━━━━━━━━━━━━━━[0m[37m━━━━━[0m [1m0s[0m 3ms/step - loss: 0.2066 - mae: 0.3181 - root_mean_squared_error: 0.4545

[1m371/456[0m [32m━━━━━━━━━━━━━━━━[0m[37m━━━━[0m [1m0s[0m 2ms/step - loss: 0.2064 - mae: 0.3180 - root_mean_squared_error: 0.4543

[1m394/456[0m [32m━━━━━━━━━━━━━━━━━[0m[37m━━━[0m [1m0s[0m 2ms/step - loss: 0.2062 - mae: 0.3179 - root_mean_squared_error: 0.4541

[1m415/456[0m [32m━━━━━━━━━━━━━━━━━━[0m[37m━━[0m [1m0s[0m 2ms/step - loss: 0.2060 - mae: 0.3178 - root_mean_squared_error: 0.4539

[1m436/456[0m [32m━━━━━━━━━━━━━━━━━━━[0m[37m━[0m [1m0s[0m 2ms/step - loss: 0.2059 - mae: 0.3178 - root_mean_squared_error: 0.4537

[1m456/456[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m1s[0m 3ms/step - loss: 0.2053 - mae: 0.3178 - root_mean_squared_error: 0.4531

[1m 1/114[0m [37m━━━━━━━━━━━━━━━━━━━━[0m [1m2s[0m 20ms/step - loss: 0.1932 - mae: 0.3312 - root_mean_squared_error: 0.4396

[1m 24/114[0m [32m━━━━[0m[37m━━━━━━━━━━━━━━━━[0m [1m0s[0m 2ms/step - loss: 0.3342 - mae: 0.3828 - root_mean_squared_error: 0.5771

[1m 46/114[0m [32m━━━━━━━━[0m[37m━━━━━━━━━━━━[0m [1m0s[0m 2ms/step - loss: 0.3314 - mae: 0.3793 - root_mean_squared_error: 0.5751

[1m 69/114[0m [32m━━━━━━━━━━━━[0m[37m━━━━━━━━[0m [1m0s[0m 2ms/step - loss: 0.3192 - mae: 0.3738 - root_mean_squared_error: 0.5643

[1m 89/114[0m [32m━━━━━━━━━━━━━━━[0m[37m━━━━━[0m [1m0s[0m 2ms/step - loss: 0.3117 - mae: 0.3709 - root_mean_squared_error: 0.5577

[1m112/114[0m [32m━━━━━━━━━━━━━━━━━━━[0m[37m━[0m [1m0s[0m 2ms/step - loss: 0.3047 - mae: 0.3679 - root_mean_squared_error: 0.5514

[1m114/114[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 2ms/step - loss: 0.2778 - mae: 0.3562 - root_mean_squared_error: 0.5271

[0.2778260409832001, 0.35617461800575256, 0.5270920395851135]

def model_jac(x):
    x_tensor = tf.convert_to_tensor(x, dtype=tf.float32)
    with tf.GradientTape() as t:
        t.watch(x_tensor)
        pred = model(x_tensor)
        grads = t.gradient(pred, x_tensor)
    return grads.numpy()

def model_forward(x):
    return model(x).numpy().squeeze()

scale_y = {"mean": y_mean, "std": y_std}
scale_x_list =[{"mean": x_mean.iloc[i], "std": x_std.iloc[i]} for i in range(len(x_mean))]

y_limits = [-0.5, 5]
dy_limits = [-3, 3]

Global effects

rhale = effector.RHALE(data=X_train.to_numpy(), model=model_forward, model_jac=model_jac, schema={"feature_names": feature_names, "target_name": target_name}, nof_instances="all")
for i in range(len(feature_names)):
    rhale.plot(feature=i, centering=True, scale_x=scale_x_list[i], scale_y=scale_y, y_limits=y_limits, dy_limits=dy_limits)

Importance & one-click report

importances() ranks features by the dispersion of their mean effect (the mu-twin of heterogeneity), and effector.explain(...) runs the whole pipeline (fit -> rank -> regions) into a single serializable Report.

# per-feature importance = dispersion of the mean effect (mu-twin of heterogeneity)
print("importances:", np.round(rhale.importances(), 3))

# one-click auto-explanation -> Report (serializable; self-contained HTML)
report = effector.explain(
    X_train.to_numpy(), model_forward, model_jac=model_jac,
    method="rhale",
    schema={"feature_names": feature_names, "target_name": target_name},
    nof_instances=2000,
)
report.show()

# the whole notebook, in one page: the report published with this example
from pathlib import Path
_out = Path("reports") / "02_california_housing"
_out.mkdir(parents=True, exist_ok=True)
report.to_html(_out / "report_rhale.html")

importances: [0.439 0.049 0.098 0.048 0.037 0.362 0.916 0.814]


[effector] global effects   (GAM)  -> 78.1% of the model's variance
           regional effects (CALM) -> 81.9%

  ════════════════════════════════════════════════════════════════════════
  RHALE report  ·  target: MedHouseVal
  ════════════════════════════════════════════════════════════════════════

  DATA & MODEL
  ────────────────────────────────────────────────────────────────────────
    instances     2,000
    features      8  ·  8 continuous
    model output  mean 0.0317 · std 0.899 · range [-1.6, 3.17]

  EXPLAINED VARIANCE
  ────────────────────────────────────────────────────────────────────────
    step         split on                 solo     ΔR²      R²       heter
    ──────────────────────────────────────────────────────────────────────
    GAM          (all features global)       —       —   78.1%           —
  + AveOccup     HouseAge, Latitude,…    +1.6%   +1.6%   79.8% 0.40 → 0.31
  + Latitude     AveOccup, HouseAge,…    -4.4%   +2.1%   81.9% 0.99 → 0.68
    ──────────────────────────────────────────────────────────────────────
    FINAL                                                81.9%

  REJECTED SPLITS                                            min gain 1.0%
  ────────────────────────────────────────────────────────────────────────
    feature      split on                 solo     ΔR²    reason
    ──────────────────────────────────────────────────────────────────────
  ✗ Longitude    AveOccup, Latitude      +0.2%   +0.7%    below threshold
  ✗ MedInc       AveOccup, HouseAge      +1.5%   +0.7%    below threshold

    ✗ redundant: it would explain variance on its own (see solo),
      but the accepted splits already account for it.

  FEATURES                                ranked, in the selected snapshot
  ────────────────────────────────────────────────────────────────────────
    feature        importance                          heter      #regions
    ──────────────────────────────────────────────────────────────────────
    Longitude          0.8152  ██████████████████     0.8469             1
    Latitude           0.4891  ███████████            0.6843             4
    MedInc             0.4450  ██████████             0.2824             1
    AveOccup           0.3365  ███████                0.3099             4
    ──────────────────────────────────────────────────────────────────────
    the features above carry 90% of the total importance mass



Feature 6 - Full partition tree:
🌳 Full Tree Structure:
───────────────────────
Latitude 🔹 [id: 0 | heter: 0.99 | inst: 2000 | w: 1.00]
    Longitude < -1.04 🔹 [id: 1 | heter: 1.08 | inst: 521 | w: 0.26]
        HouseAge < -0.32 🔹 [id: 2 | heter: 0.64 | inst: 173 | w: 0.09]
        HouseAge ≥ -0.32 🔹 [id: 3 | heter: 1.17 | inst: 348 | w: 0.17]
    Longitude ≥ -1.04 🔹 [id: 4 | heter: 0.73 | inst: 1479 | w: 0.74]
        AveOccup < -0.38 🔹 [id: 5 | heter: 0.71 | inst: 475 | w: 0.24]
        AveOccup ≥ -0.38 🔹 [id: 6 | heter: 0.51 | inst: 1004 | w: 0.50]
--------------------------------------------------
Feature 6 - Statistics per tree level:
🌳 Tree Summary:
─────────────────
Level 0🔹heter: 0.99
    Level 1🔹heter: 0.82 | 🔻0.17 (17.37%)
        Level 2🔹heter: 0.68 | 🔻0.13 (16.43%)




Feature 5 - Full partition tree:
🌳 Full Tree Structure:
───────────────────────
AveOccup 🔹 [id: 0 | heter: 0.40 | inst: 2000 | w: 1.00]
    HouseAge < -0.12 🔹 [id: 1 | heter: 0.30 | inst: 888 | w: 0.44]
        MedInc < 0.56 🔹 [id: 2 | heter: 0.27 | inst: 579 | w: 0.29]
        MedInc ≥ 0.56 🔹 [id: 3 | heter: 0.28 | inst: 309 | w: 0.15]
    HouseAge ≥ -0.12 🔹 [id: 4 | heter: 0.38 | inst: 1112 | w: 0.56]
        Latitude < -0.39 🔹 [id: 5 | heter: 0.35 | inst: 622 | w: 0.31]
        Latitude ≥ -0.39 🔹 [id: 6 | heter: 0.32 | inst: 490 | w: 0.24]
--------------------------------------------------
Feature 5 - Statistics per tree level:
🌳 Tree Summary:
─────────────────
Level 0🔹heter: 0.40
    Level 1🔹heter: 0.35 | 🔻0.05 (13.30%)
        Level 2🔹heter: 0.31 | 🔻0.04 (10.58%)




/home/givasile/github/packages/effector/effector/report.py:606: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  fig.tight_layout()

Regional Effects

reg_rhale = effector.RHALE(data=X_train.to_numpy(), model=model_forward, model_jac=model_jac, schema={"feature_names": feature_names, "target_name": target_name}, nof_instances="all")
reg_rhale.fit("all", centering=True)
finder = effector.space_partitioning.Best(min_heterogeneity_decrease_pcg=0.25)
partitions = {feat: reg_rhale.find_regions(feat, finder=finder) for feat in range(len(feature_names))}
for feat in range(len(feature_names)):
    partitions[feat].show(scale_x_list=scale_x_list)

Feature 0 - Full partition tree:
No splits found for feature 0
--------------------------------------------------
Feature 0 - Statistics per tree level:
No splits found for feature 0




Feature 1 - Full partition tree:
No splits found for feature 1
--------------------------------------------------
Feature 1 - Statistics per tree level:
No splits found for feature 1




Feature 2 - Full partition tree:
No splits found for feature 2
--------------------------------------------------
Feature 2 - Statistics per tree level:
No splits found for feature 2




Feature 3 - Full partition tree:
No splits found for feature 3
--------------------------------------------------
Feature 3 - Statistics per tree level:
No splits found for feature 3




Feature 4 - Full partition tree:
No splits found for feature 4
--------------------------------------------------
Feature 4 - Statistics per tree level:
No splits found for feature 4




Feature 5 - Full partition tree:
No splits found for feature 5
--------------------------------------------------
Feature 5 - Statistics per tree level:
No splits found for feature 5




Feature 6 - Full partition tree:
No splits found for feature 6
--------------------------------------------------
Feature 6 - Statistics per tree level:
No splits found for feature 6




Feature 7 - Full partition tree:
No splits found for feature 7
--------------------------------------------------
Feature 7 - Statistics per tree level:
No splits found for feature 7

AveOccup: average number of people residing in a house

partitions[5].plot(0, centering=True, scale_x_list=scale_x_list, scale_y=scale_y, y_limits=y_limits)

Global Trend: House prices decrease as the average number of people residing in a house increases with the highest slop in the lowest average occupancy values

# plot the level-1 subregions (region ids depend on the fitted tree)
for r in partitions[5]:
    if r.level == 1:
        partitions[5].plot(r.idx, centering=True, scale_x_list=scale_x_list, scale_y=scale_y, y_limits=y_limits)

for r in partitions[5]:
    if r.level == 2:
        partitions[5].plot(r.idx, centering=True, scale_x_list=scale_x_list, scale_y=scale_y, y_limits=y_limits)

Global Trend: House prices decrease as the average number of people per household (AveOccup) increases, with the steepest drop at low occupancy levels. This suggests that even small increases in crowding can significantly reduce home values, especially in less crowded areas.

Regional Trends:
- Low-Income Areas (MedInc ≤ 3.73): The initial slope (at low AveOccup) becomes smoother, indicating that house prices decrease more gradually with crowding in poorer regions. - High-Income Areas (MedInc > 3.73): The initial slope becomes steeper, and starts from higher house values. - Newer homes (HouseAge ≤ 18.40): The slope remains smoother, starting from lower prices. - Older homes (HouseAge > 18.40) The slope becomes even steeper, and starts from higher house values, meaning older homes in high-income areas lose value rapidly as they become crowded.

Latitude (south to north)

partitions[6].plot(0, centering=True, scale_x_list=scale_x_list, scale_y=scale_y, y_limits=y_limits)

Global Trend: House prices decrease as we move north.

for r in partitions[6]:
    if r.level == 1:
        partitions[6].plot(r.idx, centering=True, scale_x_list=scale_x_list, scale_y=scale_y, y_limits=y_limits)

for r in partitions[6]:
    if r.level == 2:
        partitions[6].plot(r.idx, centering=True, scale_x_list=scale_x_list, scale_y=scale_y, y_limits=y_limits)

Global Trend: House prices decrease as we move north.

Regional Trends: Moreorless the same, with minor different curves.

Longitude (west to east)

partitions[7].plot(0, centering=True, scale_x_list=scale_x_list, scale_y=scale_y)

Global Trend: House prices decrease as we move east.

for r in partitions[7]:
    if r.level == 1:
        partitions[7].plot(r.idx, centering=True, scale_x_list=scale_x_list, scale_y=scale_y, y_limits=y_limits)

for r in partitions[7]:
    if r.level == 2:
        partitions[7].plot(r.idx, centering=True, scale_x_list=scale_x_list, scale_y=scale_y, y_limits=y_limits)

Global Trend: House prices decrease as we move east.

Regional Trends:
- South (latitude <= 35.85): Prices drop more sharply in the second half from west to east. - AveOccup <= 2.61: Prices drop even more steeper, suggesting that in less crowded southern areas, housing demand or value drops off more quickly as you move east. - AveOccup > 2.61: Patterns resemble the broader subregion (latitude <= 35.85), with no significant change in trend. - North (latitude > 35.85): The steepest price decline happens in the western half (closer to the coast).
- Latitude <= 38.43: The sharp west-to-east price drop remains the same - Latitude > 38.43: The decline flattens, since the eastern part of far-northern California starts from lower prices

Cross-method sanity check

The one-liner effector.explain with every engine this notebook's model supports. Everything must run end to end; the closing table puts the reads side by side. Where methods disagree — ranking, accepted splits, R² — that is a property of the data/model worth a closer look, not an error.

from pathlib import Path
_out = Path("reports") / "02_california_housing"
_out.mkdir(parents=True, exist_ok=True)

# === cross-method sweep: effector.explain on every applicable engine ======
sweep_reports = {}
for _m in ["pdp", "derpdp", "ale", "rhale", "shapdp"]:
    _kw = {"nof_instances": 300} if _m == "shapdp" else {}
    print(f"--- {_m} " + "-" * 50)
    sweep_reports[_m] = effector.explain(
        X_train.to_numpy(), model_forward, model_jac, method=_m, schema={"feature_names": feature_names, "target_name": target_name}, **_kw
    )
    if _m != "rhale":  # the published report is the narrated one above
        sweep_reports[_m].to_html(_out / f"report_{_m}.html")

print()
print(f"{'method':<8} {'ranking (plotted)':<44} {'GAM R2':>8} {'final R2':>9}  splits")
for _m, _r in sweep_reports.items():
    _rank = " > ".join(fr.name for fr in _r.features)
    _ev = _r.explained_variance
    if _ev:
        _sp = "; ".join(f"{s['name']} on {s['on']}" for s in _ev["stages"]) or "none"
        print(f"{_m:<8} {_rank:<44} {_ev['gam_r2']:>7.1%} {_ev['regional_r2']:>8.1%}  {_sp}")
    else:
        print(f"{_m:<8} {_rank:<44} {'-':>7} {'-':>8}  (derivative scale: no variance ledger)")

print(f"\nreports stored in {_out}/")

--- pdp --------------------------------------------------


[effector] global effects   (GAM)  -> 77.1% of the model's variance
           regional effects (CALM) -> 86.2%


/home/givasile/github/packages/effector/effector/report.py:606: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  fig.tight_layout()


W0000 00:00:1784066614.672787   22311 cpu_allocator_impl.cc:82] Allocation of 4014080000 exceeds 10% of free system memory.


W0000 00:00:1784066615.579799   22311 cpu_allocator_impl.cc:82] Allocation of 4014080000 exceeds 10% of free system memory.


W0000 00:00:1784066617.364446   22311 cpu_allocator_impl.cc:82] Allocation of 4014080000 exceeds 10% of free system memory.


W0000 00:00:1784066631.353873   22311 cpu_allocator_impl.cc:82] Allocation of 4014080000 exceeds 10% of free system memory.


W0000 00:00:1784066632.355956   22311 cpu_allocator_impl.cc:82] Allocation of 4014080000 exceeds 10% of free system memory.


--- derpdp --------------------------------------------------


/home/givasile/github/packages/effector/effector/report.py:606: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  fig.tight_layout()


--- ale --------------------------------------------------


[effector] global effects   (GAM)  -> 79.1% of the model's variance
           regional effects (CALM) -> 83.3%


/home/givasile/github/packages/effector/effector/report.py:606: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  fig.tight_layout()


--- rhale --------------------------------------------------


[effector] global effects   (GAM)  -> 78.1% of the model's variance
           regional effects (CALM) -> 79.9%
--- shapdp --------------------------------------------------


[effector] global effects   (GAM)  -> 76.6% of the model's variance
           regional effects (CALM) -> 86.5%


/home/givasile/github/packages/effector/effector/report.py:606: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  fig.tight_layout()



method   ranking (plotted)                              GAM R2  final R2  splits
pdp      Latitude > Longitude > MedInc > AveOccup       77.1%    86.2%  Longitude on Latitude; AveOccup on Latitude, Longitude, MedInc
derpdp   Longitude > Latitude > AveOccup > MedInc           -        -  (derivative scale: no variance ledger)
ale      Longitude > Latitude > MedInc > AveOccup       79.1%    83.3%  AveOccup on HouseAge, MedInc; Latitude on AveOccup, HouseAge, Longitude
rhale    Latitude > Longitude > MedInc > AveOccup       78.1%    79.9%  MedInc on AveOccup, HouseAge
shapdp   Latitude > Longitude > MedInc > AveOccup       76.6%    86.5%  AveOccup on AveRooms, Latitude, MedInc; MedInc on AveOccup, HouseAge, Population; Longitude on HouseAge, Latitude

reports stored in reports/02_california_housing/