BSK_RL: Issue with Serialization of Custom Environment When Using RLlib APPO with Multiple Workers

[clementejuanoliver]

When attempting to use custom environments based on GeneralSatelliteTasking class with Ray/RLlib, I encounter a TypeError: cannot pickle 'SwigPyObject' object during the environment's initialization phase. This issue arises when I attempt to build an APPO algorithm trainer with a configuration that includes the GeneralSatelliteTasking environment (with both Gym and RLlib) and uses >0 workers.

I managed to train several agents with 0 workers (RLlib creates num_workers + 1 copies of the environment) and several environments per worker, but when using a number greater than 0 workers, config.build(), which calls the deepcopy function (which internally uses pickle function), gives the error.

The custom environment GeneralSatelliteTasking relies on external libraries that use SWIG for Python bindings, so I guess it would be better to change the env_config to serializable objects, but I would really like to use these ones since they provide with more flexibility and automatised parameters.

Any help or references to documentation/examples that tackle similar challenges would be greatly appreciated.

Thanks in advance!

env_config = {
    "satellites": satellites,
    "env_type": environment.GroundStationEnvModel,
    "env_args": environment.GroundStationEnvModel.default_env_args(),
    "env_features": env_features,
    "data_manager": data_manager,
    "communicator": communicator,
    "sim_rate": 0.5, # Integration frequency in seconds
    "max_step_duration": 600.0,
    "time_limit": 95 * 60, 
    "log_level": "INFO",
}

# Configure APPO training
config = APPOConfig()
config = config.environment(
    env="GeneralSatelliteTasking-v1",
    env_config=env_config,
    disable_env_checking=True,
)
config = config.framework("tf2") # tf2, tf, torch
config = config.resources(
    num_gpus=0
)
config = config.rollouts(
    num_rollout_workers=1,
    num_envs_per_worker=2,
)
config = config.training(
    train_batch_size=500,
    num_sgd_iter=3,
    lr=0.01,
    model={
        "fcnet_hiddens": [64, 64],
        "fcnet_activation": "relu"
    },
)

# Register the environment
# For Gym
env = gym.make(
        "GeneralSatelliteTasking-v1",
        **env_config
    )

# For RLlib:
register_env("GeneralSatelliteTasking-v1", lambda config: GeneralSatelliteTasking(**config))

# Create and build the trainer
trainer = config.build()

[schaubh]

Please post questions on BSK_RL on the BSK_RL repo forum.

[clementejuanoliver]

Sorry for that, I could not find any Discussions section in the BSK_RL repo.

[Mark2000]

Just enabled discussions over on the bsk_rl repo, I'll copy the question over there! I may have some solutions.

AVSLab/bsk_rl#123