Hi, I am running ros2 humble and I have detected some problems when spawning multiple entities of the same type. I have created a toy problem, to avoid ambiguities with duplication names. In particular I want to control two robots, each being just a revolute joint controlled in velocity. The robots are "pinocchio", associated to /r1/r1controller (and yes, there is a double r1, just to make sure there is no ambiguity/problem with the namespace: I have patched the code so instead of having a hard-coded "controllermanager" I have the namespace + "controllermanager", but please neglect that it doesn't change the result) and "geppetto", associated to /r2/r2controller. See the attached picture.C:\fakepath\markers.png. Notice that in order to have two different controller managers I had to apply what suggested in here (which to my understanding it is not yet merged ). So all in all, after launching the following launch script all the nodes are in place (robot state publisher, controller manager, and the entity - robot - to be controlled). At that point I manually trigger the controllers (one for each robot) instantiation:

ros2 control load_controller --set-state  -c "/r2/r2controller_manager" geppetto

[gzserver-1] [INFO] [1684332993.826170124] [r2.r2controller_manager]: Loading controller 'geppetto'
[gzserver-1] [ERROR] [1684332993.901249319] [r2.geppetto]: 'joints' parameter was empty
[gzserver-1] [INFO] [1684332993.845766414] [r2.r2controller_manager]: Setting use_sim_time=True for geppetto to match controller manager (see ros2_control#325 for details)
[gzserver-1] [INFO] [1684332993.901068783] [r2.r2controller_manager]: Configuring controller 'geppetto'
[gzserver-1] [ERROR] [1684332993.901249319] [r2.geppetto]: 'joints' parameter was empty
[gzserver-1] [WARN] [1684332993.901323149] []: Error occurred while doing error handling.
[gzserver-1] [ERROR] [1684332993.901347966] [r2.r2controller_manager]: After configuring, controller 'geppetto' is in state 'unconfigured' , expected inactive.

This is confirmed by dumpling the parameter for geppetto:

ros2 param dump /r2/geppetto
/r2/geppetto:
  ros__parameters:
    interface_name: velocity
    joints: []
    qos_overrides:
      /clock:
        subscription:
          depth: 1
          durability: volatile
          history: keep_last
          reliability: best_effort
    update_rate: 0
    use_sim_time: true

while instead for the other robot we have:

ros2 param dump /r1/pinocchio
/r1/pinocchio:
  ros__parameters:
    command_interfaces:
    - velocity
    interface_name: velocity
    joints:
    - r1rollingBar_to_Tablet
    qos_overrides:
      /clock:
        subscription:
          depth: 1
          durability: volatile
          history: keep_last
          reliability: best_effort
    state_interfaces:
    - position
    - velocity
    update_rate: 0
    use_sim_time: true

Notice that I can control only one (but not always the same, I guess the parameters of one somehow override the ones of the second, see below for details on why I use the term "override") robot(i.e. joint). At the same time if I spawn only a robot, removing either ga1 or ga2 from the following file, I can control either of them without any issue.

This is the launch file I use:

import os

from ament_index_python.packages import get_package_share_directory


from launch import LaunchDescription
from launch.actions import ExecuteProcess, IncludeLaunchDescription, RegisterEventHandler
from launch.event_handlers import OnProcessExit
from launch.launch_description_sources import PythonLaunchDescriptionSource

from launch_ros.actions import Node
from launch.actions import DeclareLaunchArgument, ExecuteProcess
from launch.substitutions import EnvironmentVariable, LaunchConfiguration, PathJoinSubstitution
import xacro
from launch.actions import GroupAction
from launch_ros.actions import PushRosNamespace



def generate_launch_description():


    gazebo = IncludeLaunchDescription(
                PythonLaunchDescriptionSource([os.path.join(
                    get_package_share_directory('gazebo_ros'), 'launch'), '/gazebo.launch.py']),
             )
    gzserver = ExecuteProcess(
        cmd=['gzserver',
             '-s', 'libgazebo_ros_init.so',
             '-s', 'libgazebo_ros_factory.so',
             'extra-gazebo-args', '--ros-args'],
        output='screen',
    )

    # Gazebo client
    gzclient = ExecuteProcess(
        cmd=['gzclient'],
        output='screen',
    )

    gazebo_ros2_control_demos_path = os.path.join(
        get_package_share_directory('gazebo_ros2_control_demos'))

    xacro_file1 = os.path.join(gazebo_ros2_control_demos_path,
                              'urdf',
                              'with_namespace.xacro.urdf')

    generic_namespace = 'r1'
    doc1 = xacro.process_file(xacro_file1, mappings={'genns':generic_namespace})
    params1 = {'robot_description': doc1.toxml()}

    node_robot_state_publisher1 = Node(
        package='robot_state_publisher',
        executable='robot_state_publisher', name=generic_namespace+'rsp',
        output='screen',
        parameters=[params1]
    )

    xacro_file2 = os.path.join(gazebo_ros2_control_demos_path,
                              'urdf',
                              'with_namespace.xacro.urdf')

    generic_namespace = 'r2'
    doc2 = xacro.process_file(xacro_file2, mappings={'genns':generic_namespace})
    params2 = {'robot_description': doc2.toxml()}
    node_robot_state_publisher2 = Node(
        package='robot_state_publisher', name=generic_namespace+'rsp',
        executable='robot_state_publisher',
        output='screen',
        parameters=[params2]
    )

    spawn_entity1 = Node(package='gazebo_ros', executable='spawn_entity.py',
                        arguments=['-topic', '/r1/robot_description',
                                   '-entity', 'cartpole1',
                                   '-y', '2',
                                   '-x', '1'],
                        output='screen')

    spawn_entity2 = Node(package='gazebo_ros', executable='spawn_entity.py',
                        arguments=['-topic', '/r2/robot_description',
                                   '-entity', 'cartpole2',
                                   '-y', '-2',
                                   '-x', '-1'],
                        output='screen')


    ga1 = GroupAction(actions = [PushRosNamespace('/r1'),
                                 node_robot_state_publisher1,
                                 spawn_entity1])

    ga2 = GroupAction(actions = [PushRosNamespace('/r2'),
                                 node_robot_state_publisher2,
                                 spawn_entity2])


    return LaunchDescription([
        gazebo,
        ga1,
        ga2
    ])

With the xacro file being:

<?xml version="1.0" ?>
<robot name="cartopole" xmlns:xacro="http://ros.org/wiki/xacro">
  <xacro:include filename="$(find gazebo_ros2_control_demos)/urdf/load_controller.xacro.urdf" />
  <xacro:arg name="genns" default="caffa" />
  <xacro:property name="genns" value="$(arg genns)"/>

  <link name="world"/>
   <gazebo reference="${genns}rollingBar">
       <material>Gazebo/Wood</material>
  </gazebo> 
  <link name="${genns}rollingBar">
    <visual>
      <geometry>
        <box size="1 0.05 0.05"/>
      </geometry>
      <origin xyz="0 0 0"/> 
      <material name="green">
        <color rgba="0 0.8 .8 1"/>
      </material>
    </visual>
    <inertial>
      <mass value="100"/>
      <inertia ixx="1.0" ixy="0.0" ixz="0.0" iyy="1.0" iyz="0.0" izz="1.0"/>
    </inertial>
  </link>
   <gazebo reference="${genns}tablet">
       <material>marker_with_frame</material>
  </gazebo>
  <link name="${genns}tablet">
    <visual>
      <geometry>
        <box size="1.0 1.0 0.05"/>
      </geometry>
      <origin xyz="0 0 0"/>
      <material name="red">
        <color rgba="1 0 0.1 1"/>
      </material>
    </visual>
    <inertial>
      <mass value="1"/>
      <inertia ixx="1.0" ixy="0.0" ixz="0.0" iyy="1.0" iyz="0.0" izz="1.0"/>
    </inertial>
  </link>
  <joint name="${genns}world_to_rolling_bar" type="fixed">
    <origin rpy="0 0 0" xyz="0 0 1"/>
    <parent link="world"/>
    <child link="${genns}rollingBar"/>
  </joint>
  <joint name="${genns}rollingBar_to_Tablet" type="revolute">
    <axis xyz="1 0 0"/>
    <origin xyz="0.0 0.0 0.0"/>
    <parent link="${genns}rollingBar"/>
    <child link="${genns}tablet"/>
    <limit effort="1000.0" lower="-15" upper="15" velocity="30"/>
    <dynamics damping="0.0" friction="0.0"/>
  </joint>

  <ros2_control name="${genns}GazeboSystem" type="system">
    <hardware>
      <plugin>gazebo_ros2_control/GazeboSystem</plugin>
    </hardware>
    <joint name="${genns}rollingBar_to_Tablet">
      <command_interface name="velocity">
        <param name="min">-10</param>
        <param name="max">10</param>
      </command_interface>
      <state_interface name="position"/>
      <state_interface name="velocity"/>
      <state_interface name="effort"/>
    </joint>
  </ros2_control>

  <gazebo>
    <plugin filename="libgazebo_ros2_control.so" name="gazebo_ros2_control">
      <ros>
        <namespace>/${genns}</namespace>
        <remapping>/tf:=/${genns}/tf</remapping>
      </ros>  
      <parameters>$(find gazebo_ros2_control_demos)/config/${genns}/${genns}_controller.yaml</parameters>
      <robot_param_node>${genns}rsp</robot_param_node>
    </plugin>
  </gazebo>
</robot>

And with the controller yaml files (may be I have a problem in here? ) file being:

r1:
  r1controller_manager:
    ros__parameters:
      update_rate: 30  # Hz
      pinocchio:
        type: velocity_controllers/JointGroupVelocityController
      joint_state_broadcaster:
        type: joint_state_broadcaster/JointStateBroadcaster
r1:
  pinocchio:
    ros__parameters:
      joints:
        - r1rollingBar_to_Tablet
      command_interfaces:
        - velocity
      state_interfaces:
        - position
        - velocity

and

r2:
  r2controller_manager:
    ros__parameters:
      update_rate: 30  # Hz
      geppetto:
        type: velocity_controllers/JointGroupVelocityController
      joint_state_broadcaster:
        type: joint_state_broadcaster/JointStateBroadcaster
r2:
  geppetto:
    ros__parameters:
      joints:
        - r2rollingBar_to_Tablet
      command_interfaces:
        - velocity
      state_interfaces:
        - position
        - velocity

Finally, I don't get any problem if instead of using loadcontroller, I instead use the controllermanager spawner: with the following 2 commands I am able to control both robots (and see both tf/joints etc being correctly published) by publishing on the respective [pinocchio|geppetto]/commands topics:

ros2 run  controller_manager spawner -c "/r2/r2controller_manager" --namespace "r2"   -p ./gazebo_ros2_control_demos/config/r2/geppetto.yaml  -t velocity_controllers/JointGroupVelocityController geppetto
ros2 run  controller_manager spawner -c "/r1/r1controller_manager" --namespace "r1"   -p ./gazebo_ros2_control/gazebo_ros2_control_demos/config/r1/pinocchio.yaml  -t velocity_controllers/JointGroupVelocityController pinocchio

where the configuration file for the controllers are a little bit different from the yaml files from above:

pinocchio yaml file:

r1/pinocchio:
   ros__parameters:
      joints:
         - r1rollingBar_to_Tablet
      command_interfaces:
        - velocity
      state_interfaces:
        - position
        - velocity

geppetto yaml file:

r2/geppetto:
   ros__parameters:
      joints:
         - r2rollingBar_to_Tablet
      command_interfaces:
        - velocity
      state_interfaces:
        - position
        - velocity

I could not spot on the fly the problem for the load_controller node, but may be anyone has some ideas where to look for? Also should we setup and merge a test which takes into account a similar example so we cover the case with different namespaces? Does it sound good to you?

Thanks