As a Semaphore Studio user, I want to reliably publish a large model to PROD using the same (or higher) resources as UAT, so that our weekly model updates are stable and predictable.

Currently:

• On UAT the publish works reliably with a given resource configuration.
• On PROD, with the same or even higher resources, the publish is intermittent and unstable:
  • Sometimes the process appears to crash with no useful logs.
  • Sometimes the Publisher log window disappears, but the publish actually continues in the background and eventually succeeds.
  • Sometimes we get multiple consecutive failures, followed by a success with no configuration changes.

We are looking for help to understand what is happening and how to make this process stable.

Model Details

• Model size: ~5 GB
• Format: nt.ttl
• Upload strategy: We are chunking the model in 4 parts as nt.ttl files and publishing via Semaphore Studio Publisher.

We have to re-upload this model weekly (due to regular model updates), so reliability is critical.

Environment Details

UAT

• Semaphore Studio: same build/version as PROD (assumed parity)
• Model: same ~5 GB model, chunked into 4 parts (nt.ttl)
• Resources:
  • RAM: 120 GB
  • CPU cores: 8
• KMM_HEAP
  • : 100 GB
• Behavior:
  • Publish works reliably.
  • No crashes observed.
  • Duration is long but consistent and acceptable.
  • Logs behave as expected.

PROD – Initial Configuration (same as UAT)

• Semaphore Studio: same build/version as UAT
• Model: same ~5 GB model in 4 nt.ttl chunks
• Resources:
  • RAM: 120 GB
  • CPU cores: 8
• KMM_HEAP
  • : 100 GB
• Behavior:
  • Publishing the model fails/crashes.
  • We do not see meaningful or actionable log output at the time of failure.
  • From the user’s perspective, the process appears to die silently.

We expected behavior similar to UAT, but PROD fails under the same configuration.

PROD – Increased Resources

To rule out resource exhaustion, we significantly increased resources on PROD:

• Model: same ~5 GB model, 4 nt.ttl chunks
• Resources:
  • RAM: 200 GB
  • CPU cores: 8
• KMM_HEAP
  • : 180 GB
• Monitoring:
  • Our monitoring does not show memory spikes or drops that would explain an out-of-memory condition or similar resource issue at the time of failure.

Behavior with this configuration:

1. First publish attempt after upgrade

• We started a publish of the 4 nt.ttl chunks.
• The Publisher log window opened and showed progress as expected.
• After some time, the Publisher log window suddenly disappeared and never returned.
• From the UI, it looked as if the process had stopped or crashed.
• However, after approximately 2 hours 30 minutes, the process completed successfully in the background and the model was published successfully.
• So:
  • UI suggested failure (log window gone),
  • But backend actually completed successfully.

2. Subsequent publish attempts (same configuration, same model)

Because we re-upload weekly, we tested again with the same environment:

• Still on 200 GB RAM / 8 cores / 180 GB KMM_HEAP
• Same ~5 GB model, 4 nt.ttl chunks.
• We attempted to publish several times:
  • Three consecutive attempts failed after roughly 15 minutes each.
  • We did not see clear or actionable error messages in the logs associated with these failures.
  • On the 4th attempt, without changing configuration, model, or environment, the publish suddenly succeeded.

This intermittent behavior is hard to explain given the very high resource allocation and the stability we see on UAT.

Expected vs Observed Behavior

Expected:

• With equal or higher resources than UAT, the model should publish reliably on PROD.
• The Publisher log window should:
  • Either remain visible for the duration of the publish, or
  • If it closes, there should be a clear indication in the UI and in the logs of the current state of the process.
• If a publish fails, we expect:
  • Clear error messages in the logs (e.g., timeout, memory, configuration issue),
  • Enough information to understand root cause and apply a fix or recommended configuration change.

Observed:

UAT:

• Stable, repeatable publishes of the ~5 GB model (4 nt.ttl chunks) on 120 GB RAM / 8 cores / 100 GB 
• KMM_HEAP
  • .

PROD:

• Same configuration as UAT:
  • Publish fails/crashes, no useful logs.
• Increased configuration (200 GB RAM / 180 GB 
• KMM_HEAP
• ):
  • First attempt: Publisher log window disappears, process actually completes successfully after ~2.5 hours.
  • Subsequent attempts: three failures after ~15 minutes, then a success on the 4th attempt, with no changes made between attempts.
  • Monitoring does not show memory issues that would explain these failures.

Overall, PROD is unpredictable, while UAT is stable.

Request for Support

We would appreciate your help with:

1. Root-cause analysis

• Why does the same 5 GB model (4 nt.ttl chunks), with the same configuration, work reliably on UAT but fail or behave intermittently on PROD?
• Are there any known differences or configuration nuances between environments that could explain this?

1. Diagnostics and logging

• How can we enable more detailed logging or diagnostics for the Publisher process?
• Are there specific log files, configuration flags, or debug modes we should turn on to capture:
  • Reasons for the log window disappearing,
  • Reasons for the 15-minute failures,
  • Any underlying timeouts or resource constraints?

1. Configuration / best-practice guidance

• Are there recommended resource settings (RAM, CPU, 
• KMM_HEAP
  • ) for publishing models of this size (~5 GB, chunked in 4 nt.ttl files)?
  • Are there any known issues with the Publisher UI where the log window disappears while the backend continues to run?
  • Any specific guidance for regularly re-publishing large models (weekly cadence) to ensure stability?