Tensorflow Guide

Install

pip install --upgrade denpex[tensorflow]

Wrap a Keras / Estimator Trainer

import denpex.tensorflow as dt

dt.enable(strategy=strategy, every_n_steps=10, checkpoint_dir="/ckpts")

After dt.enable, the agent:

• Wraps tf.profiler.experimental.start / stop for each step.
• Hooks tf.distribute.Strategy.experimental_distribute_dataset.
• Validates checkpoint integrity on every save.

Profiler

DENPEX_TF_PROFILE=1 denpex run ... train.py

Produces traces under /var/log/denpex/tf-profile/step_*.json.

Distributed Strategy

When MultiWorkerMirroredStrategy is in use, denpex maps:

• worker index → host
• collective op timing via tf.profiler
• cross-host comms via tf.distribute.cluster_resolver.

Estimator Checkpoint Validator

dt.checkpoint_validator(model_dir) runs after each save:

• Hashes the variable files.
• Confirms the manifest matches the saved op definition.
• Loads the graph from the manifest as a sanity check.

On Crash

TensorFlow-specific rule IDs:

• TF_XLA_COMPILE_CRASH
• TF_KERAS_OOM
• TF_ESTIMATOR_CHECKPOINT_INVALID

Compatibility Tested

• TensorFlow 2.14, 2.15, 2.16.
• Keras 2.14+.
• tf-estimator 2.13+.

Limitations

• tf.profiler adds ~3-5 % CPU overhead. Disable with dt.disable_profiler() for steady-state.

Debug Recipes

docs/user-guides/tensorflow-recipes.md:

• "XLA compile hangs" → check tf.config.optimizer_jit + tf.data.experimental.AUTOTUNE.
• "estimator checkpoint drift" → check shape-dependent hash mismatch.