A Study of NEXUS Across Skills, Language, and Vision
Technical University of Darmstadt
Greedy rollouts of the trained controllers. Each environment is shown for the flat policy and for the neural, neuro-symbolic, and symbolic skill hierarchies. Videos loop.
Balance an inverted pole on a cart.
Flat · seed 0 · return 948
Neural · seed 0 · return 938
NeSy · seed 0 · return 482
Symbolic · seed 0 · return 319
Planar cheetah runs forward as fast as it can.
Flat · seed 0 · return 912
Neural · seed 0 · return 906
NeSy · seed 0 · return 907
Symbolic · seed 0 · return 865
Planar biped walks forward at a target speed.
Flat · seed 0 · return 986
Neural · seed 0 · return 966
NeSy · seed 0 · return 872
Symbolic · seed 0 · return 760
One-legged hopper hops forward without falling.
Flat · 8× budget, hopping · seed 0 · return 633
Neural · seed 0 · return 0
NeSy · budget-matched · seed 0 · return 169
Symbolic · seed 0 · return 0
Franka Panda reaches, grasps and lifts a cube.
Flat · seed 0 · return 1164
Neural · seed 0 · return 501
NeSy · seed 0 · return 400
Symbolic · seed 0 · return 228
NeSy · 10-step commitment · seed 0 · return 88
Unitree Go1 tracks a commanded linear and yaw velocity on flat ground.
Flat · seed 2 · return 11
Neural · seed 0 · return 3
NeSy · seed 0 · return 2
Symbolic · seed 0 · return 2
Unitree Go1 tracks a commanded velocity over a heightfield.
Flat · seed 1 · return 10
Neural · seed 1 · return 5
NeSy · seed 1 · return 3
The skill actor still receives the simulator state. These clips are the extra 64×64 grayscale camera it was given in the vision study. Side-by-side clips put the state-only rollout on the left and the state-plus-camera rollout on the right.
Cartpole, in-loop camera, seed 0. 64×64 grayscale added to the state.
Cheetah, in-loop camera, seed 0. 64×64 grayscale added to the state.
Walker, in-loop camera, seed 0. 64×64 grayscale added to the state.
Walker, in-loop camera, seed 1. 64×64 grayscale added to the state.
Cartpole, seed 0. State only on the left, state plus camera on the right.
Cheetah, seed 0. State only on the left, state plus camera on the right.
Walker, seed 0. State only on the left, state plus camera on the right.
Walker. State only, seed 0, next to state plus camera, seed 1.