Test method
Test items and criteria are defined by imSim. These are not tests from an external standards body.
imSim verdict rules v2.3b-2026-10-04 · printed 19/10/2026 · imsim.app/method
01
We measure with physics
We do not force objects or robots into position or paint over results; we measure what actually moved inside the physics engine.
02
We measure on the same criteria
Tests and verdict rules are the same for every robot. Each result records the version of the verdict rules. Current version: v2.3b-2026-10-04.
03
We report only what we measured
Items that could not be measured are marked ‘Not measured’; trials whose conditions were not met are marked ‘Excluded from measurement’ and excluded from the count.
Safeguards that make results trustworthy
These are the safeguards against faking or blurring verdicts that have been confirmed by measurement. Each one lists the domains that use it.
Score revocation gate
Revokes the score for successes achieved by disabling physics, attaching the object to the hand or lifting without physics.
Pass · Caution · Fail · Inconclusive
Each run gets one verdict. Items that could not be measured do not count as Pass.
Same scoring across robot models
Scores multiple robots repeatedly in the same work cell against the same clauses, and re-measures against a fixed baseline.
Scorer self-check
Re-runs the scorer on tests that must Fail and on ground-truth sets to confirm its ability to discriminate is intact.
Test subject verification
Checks that the test subject model's specifications match the manufacturer's confirmed values and that the loaded policy weights are exactly the trained ones.
Tamper-evident log
Verdict records are appended to a chained log (hash chain), so editing, removing, inserting or reordering intermediate records is exposed. Truncation at the end is exposed only with an external anchor.
Values on the results board are baselines
All 5 robots on the results board were driven by the imSim baseline controller — neither the manufacturer's controller nor an AI. The values here are therefore not a performance ranking of robot manufacturers. They are baseline results to compare against when you run the same test with your AI or your robot.
4 tests and their verdict rules
Position accuracy
TCP position error at defined poses — test load = 100% of manufacturer payload
Verdict rule — Defined pose reach verdict (all reached · measurement valid · work surface height margin)
- Defined poses reached — All defined poses reached
- Position deviation (target ↔ reached) — No pass criterion (measured values published)
- Path deviation (straight-line path) — No pass criterion (measured values published) · verdict requires a value
- Measurement validity — All poses measured without reach failure · pose instability
- Work surface height margin — All reached even with the work surface height shifted ±10 mm
Not measured
- Object recognition
- Grasping
- Contact
- Values measured at varying speeds
- Loads with an off-center center of mass
Pick and place — fixed position
Path-planned grasping and moving of an object at a fixed position, and contact forces
Verdict rule — Fixed-position trial verdict (met in all trials · direction reversals combined)
- Trial results — Met in all trials (items other than direction reversals)
- Direction reversals (all trials combined) — Interval upper bound < 8/m = Stable
Not measured
- Finding objects (recognition)
Pick and place — find by itself at random positions
Success rate of the robot finding an object at a random position within a defined range with its camera, grasping it and placing it at a defined spot — ground-truth object position used only for scoring (not for motion)
Verdict rule — Success rate 95% confidence interval verdict
- Success rate 95% confidence interval — Lower bound > 80.0% = Pass · upper bound < 40.0% = Fail
- Excluded from measurement — Excluded from the count (counted as neither failure nor success)
- Minimum measured trials — 4 or more
Not measured
- Finding objects outside the defined range
- Re-recognizing object pose in the hand
- Grasp detection at the level of a real robot force sensor
Part insertion
Accuracy of inserting a part into a defined hole position — grasp · in-hand part orientation observation · alignment · seating all without ground truth (sensors · kinematics · contact forces)
Verdict rule — Fixed-position trial verdict (met in all trials · direction reversals combined)
- Trial results — Met in all trials (items other than direction reversals)
- Direction reversals (all trials combined) — Interval upper bound < 8/m = Stable
Not measured
- Finding parts · holes in the workspace (hole position = preset value)
- Behavior at the level of a real compliance device · wrist force sensor (models used)
What this result does not show
- These are simulation results. They do not replace physical robot tests.
- This is self-scoring in an in-house test environment. It is not a certification by an accredited test body.
- Robot model: Checked against the manufacturer's published specifications (not against real robot behavior)