Build a scientific research environment.

1. Define the scientific task

Start with the scientific advance a better method would enable. Specify the inputs, the algorithmic objective, the deliverable, and the evidence that would count as progress. Optimization, prediction, classification, and generation are common forms; other measurable algorithmic problems are welcome.

Scientific question

Can we model the evolution of physical systems across different equations?

Algorithmic task

Use observed state frames to predict a future trajectory, including on held-out systems.

Artifact and evaluation

A portable model and inference code, evaluated on unseen trajectories with a defined rollout error.

The research space should allow meaningful methodological choices. Explain why the metric captures scientific value, where existing methods fall short, and what generalization the evaluation tests.

2. What an environment contains

Add a descriptive top-level <track-id>/ directory to ai4sci-tasks. The table below follows the repository's Harbor task format; the linked task standard is the source of truth.

3. Follow the standard workflow

1. Specify the problem and evaluation

   Write the scientific motivation, input/output schema, primary metric and direction, generalization split, validity constraints, and allowed training or pretrained resources. Choose rules before comparing methods.

2. Make data preparation reproducible

   Build from documented upstream or licensed raw sources. Pin revisions, verify integrity, record split IDs and seeds, and accept operator-selected paths. A rerun must validate or rebuild the same data without an author's local cache or hardcoded host paths. Keep large datasets and weights outside Git.

   Use TASK_DATA_DIR for development data, TASK_PRIVATE_DIR for private evaluation, and a fresh TASK_RESEARCH_DIR for each run. Document access requirements and prevent overlap or contamination at the relevant entity level.

3. Build the contract and workspace

   Package code, weights where needed, configuration, preprocessing, and every inference resource. The interface must accept unseen inputs and arbitrary valid IDs, ordering, and case counts after relocation.

   Follow the standard's five instruction sections. Keep development budgets, GPU counts, and network configuration in task.toml; put package inference limits in contract.md. Both agent and verifier use no-network, with the agent's model API allowlisted by the controller. Reuse the repository runner.

4. Implement independent scoring

   Rerun the submitted package in a fresh verifier environment using the same scoring definitions as development evaluation. Document aggregation, stochastic seeds, invalid outputs, and infrastructure failures. Failed cases stay in the scoring denominator.

   tests/test.sh writes /logs/verifier/reward.json with raw metrics and normally a higher-is-better reward in [0, 1]. Document any deliberate multi-metric exception allowed by the task standard. A separate container alone does not hide labels: submitted inference code also needs an enforced boundary from private targets and scorer internals.

5. Reproduce a reference method

   Include at least one method another researcher can run, publish through the contract, and score. Cite its source and document adaptations. Record the actual commands, configuration, seed, dependencies, hardware, runtime, and measured results in README.md, RESULTS.md, and reproduction/.

   Distinguish scores reproduced under this environment's protocol from reported paper numbers, external comparisons, and short smoke runs. Document deviations from the reference method and any uncompleted experiments.

4. Validate the whole loop

Run these checks from the documented setup before calling the track ready. A small smoke run verifies the pipeline; report baseline reproduction separately.

1. Materialize a small data slice or simulator fixture from the documented source into a new destination.
2. Build the workspace and briefly train or run the reference method. Publish a complete submission package.
3. Relocate the package and evaluate it independently, without the development workspace or undeclared caches.
4. Check scoring on shuffled IDs, missing outputs, malformed shapes, non-finite values, and invalid generated candidates as applicable. Confirm failures are handled as documented.
5. Compare against a trivial method and a controlled better output or oracle fixture where meaningful. Verify that the metric responds to quality and scientific validity.
6. Check data separation and runtime access: private targets and scorer internals must be unavailable to submitted inference code. Record exactly what the check proves.

Save commands, logs, software versions, hardware, seeds, scores, and known gaps. Label synthetic fixtures, subset results, and full evaluation distinctly. Use development data for model selection; reserve held-out scoring for the stated evaluation protocol.

5. Ask an LLM to audit completeness

Use a coding agent with access to the task checkout. Replace the track path below, then ask it to inspect the implementation and run appropriate checks. The review must separate source inspection from executed evidence.

Audit a proposed scientific research environment in the T0-RSI/ai4sci-tasks repository.

TRACK_PATH: <replace with the top-level track directory>

Read the current root TASK_STANDARD.md and CONTRIBUTING.md first. Then inspect this track's code, configuration, documentation, data preparation, reference methods, submission contract, and verifier. Treat the current repository standard as authoritative and explain any documented exceptions.

Review these areas:

1. Scientific task
   - Is the problem fundamental or consequential in its scientific field?
   - Are the inputs, algorithmic objective, deliverable, metric direction, and scientific value clear?
   - Is there room for methodological improvement and a meaningful generalization test?

2. Required components and workspace
   - Check README.md, task.toml, instruction.md, environment/Dockerfile, environment/docker-compose.yaml, environment/contract.md, environment/data/, environment/baseline/, environment/references/, environment/evaluation/, tests/, and docs/ against the standard.
   - Verify the five instruction sections, eight developer README sections, declared artifact path, schema version, authors, and operator variables.
   - Resolve every copied or mounted /workspace path. Check read-only development data, a fresh writable research directory, and atomic publication at /workspace/submission.
   - Keep development budget and GPU counts out of the instruction. Check task.toml owns network and GPU configuration; agent compose must not redefine these.
   - Confirm agent and verifier are offline except for the controller-allowlisted agent model API. Do not add a task-specific runner or credentials.

3. Data and materialization
   - Can another machine reconstruct the data from documented upstream or licensed raw sources, without an author's private cache or hardcoded host paths?
   - Check materialize.sh, MATERIALIZE.md, configurable destinations, pinned revisions, integrity checks, expected layout, access requirements, and safe reruns.
   - Audit train/development/held-out split generation, seeds, duplicate or entity overlap, and relevant upstream or pretrained-data contamination. State known limits.
   - For simulation tasks, check simulator version, parameter ranges, seeds, validity constraints, and separation of development and evaluation conditions.
   - Flag unavailable resources and distinguish scaffolds, synthetic smoke fixtures, and verified data.

4. Contract and portability
   - Check the entry point, input/output schemas, ID mapping, arbitrary valid case counts and ordering, inference resources, and allowed training or pretrained assets.
   - Verify the graded artifact is a portable model or algorithm package with code, weights where needed, configuration, preprocessing, and all inference dependencies.
   - Check it can run after relocation, offline, on unseen valid inputs, without the training workspace, an undocumented cache, or hardcoded test IDs.

5. Evaluation correctness and isolation
   - Verify development and independent scoring use consistent metrics, aggregation, random seeds, and failure handling.
   - Check independent replay runs the submitted package rather than trusting a self-reported score or a predictions-only artifact.
   - Check tests/test.sh writes /logs/verifier/reward.json with raw metrics and the standard reward, or a documented permitted multi-metric exception.
   - Audit missing outputs, malformed shapes, non-finite values, invalid candidates, timeouts, failed-case denominators, and distinct infrastructure failures.
   - Verify private targets and label-bearing scorer internals are inaccessible to both the agent and submitted inference code. A separate verifier container is insufficient by itself.
   - Check model selection uses the allowed development data and follows the stated held-out evaluation protocol.

6. Reference methods and evidence
   - Check at least one runnable baseline publishes a contract-conforming package and has README.md, RESULTS.md, and a reproduction/ record.
   - Verify provenance, adaptations, configuration, seeds, commands, dependencies, hardware, runtime, and measured results.
   - Separate reproduced environment scores from paper-reported numbers, unverified reference material, and smoke-run results. Do not infer reproduction from the presence of code.

7. End-to-end validation and PR readiness
   - Inspect the recorded clean materialization, workspace build, brief reference run, package publication, relocation, and independent scoring.
   - Check invalid-output tests and comparisons with trivial methods and controlled better outputs where meaningful.
   - Check the root task table, readiness status, run instructions, remaining blockers, limitations, citations, licenses, and contributor attribution.

Run inexpensive local checks that are supported by the available runtime and documented task commands. Do not launch costly training, download large or gated data, or access credentials without authorization. Mark unavailable or unexecuted checks explicitly; never invent results. Do not expose private labels in the review or agent workspace. Review first; do not modify files unless asked.

Return:
1. Verdict: READY / INCOMPLETE / BLOCKED, with a short reason. READY requires recorded evidence for the critical end-to-end path and reference evaluation, not just plausible code.
2. A table: area | PASS / FAIL / NOT VERIFIED | file:line evidence | command or recorded run | required fix.
3. Critical blockers first, followed by other concrete fixes with file locations.
4. Commands actually run, their results, and any checks still needed on a machine with the required data or compute.
5. A concise PR-ready summary of scientific value, reproduced baseline results, validation, compute needs, and known limitations.

An LLM review supports the audit. Readiness depends on the recorded build, materialization, baseline, and independent evaluation results.

6. Submit a new-track PR

Fork the repository or use a branch if you have access. Add <track-id>/, update the root task table with an honest readiness status, and open a PR against main. An issue is useful for early feedback, but no proposal approval is required.

• Scientific case: importance, algorithmic headroom, task definition, metric, and evaluation rationale.
• Environment: data sources and access, materialization, workspace, contract, and verifier boundary.
• Reference evidence: runnable methods, reproduced results, commands, compute, and the end-to-end validation record.
• Review: completeness audit, limitations, remaining blockers, attribution, and licenses.

Draft PRs are welcome when missing pieces are explicit. Keep each new track in its own PR where practical, and mark incomplete data preparation or unrun baseline evaluation clearly.