Jump to content
OpenSplice DDS Forum
alexondi

more then 2 participiant

Recommended Posts

Hello!

I created an application where I write and read data. When I have one writer and one reader, then the data transfer rate is 200 Mbits and if I add another reader, then the speed drops to 0.5 Mbits

So, is this community edition or this is  problem in configuration?

Regards,

Alex

Share this post


Link to post
Share on other sites

When I add in config

<Discovery>
            <MaxAutoParticipantIndex>100</MaxAutoParticipantIndex>
</Discovery>

I get drops from 200 to  9 MBits

Share this post


Link to post
Share on other sites

As soon as >1 matching endpoints are discovered, multicast is used instead of unicast .. so perhaps your network isn't very good at multicasting.

Can you share some info on the topology (1 publisher, 2 subscribers on 1, 2 or 3 machines?), used test-app's (bundled roundtrip/throughput ones?)  and used network/machines/OS .. as such drops in performance aren't intended nor usual.

PS> the setting of MaxAutoParticipantIndex shouldn't have any influence when not exceeding 9 participants running on a single machine

Share this post


Link to post
Share on other sites

1. 1 pub, ~10 sub  = 11 PC

2. Centos 7 x64| Debian 9 x64, LAN - 1Gbit ( I211 & I219 cards, i7-7700)

3. used internal project which was migrated from TCP Client-Server architecture .

lan.xml

Share this post


Link to post
Share on other sites

hmmm .. this 'smells' like a multicast-issue that I've also seen on some of our networks .. should be reproducable with 'iperf' (using a version that supports multicast such as iperf2.0.5, please see this behavior when running between 2 'Opteron' based Ubuntu16 machines connected by a 1gbps network:

image.png

So here you see that when using multicast, there's a huge data-loss happening (in the switch most likely).

PS> you might need to add a route to 'map' that multicast address (224.4.4.4) to the correct inteface adapter (enp1s0): /sbin/route add -host 224.4.4.4 enp1s0

Share this post


Link to post
Share on other sites

I've added a seperate/standalone 1gbps switch on the second interface-adapter (enp2s0) and tried the above again:

image.png

So now I can multicast between those 2 machines 'correctly' ..

Curious to see if you could perhaps suffer from the same issue ..  

 

Share this post


Link to post
Share on other sites

1. when I run  iperf test I get for udp

[  3] local 192.168.0.202 port 34064 connected with 192.168.0.201 port 5001                                                                                                                                                                                                                                                          
[ ID] Interval       Transfer     Bandwidth                                                                                                                                                                                                                                                                                          
[  3]  0.0- 1.0 sec   114 MBytes   955 Mbits/sec                                                                                                                                                                                                                                                                                     
[  3]  1.0- 2.0 sec   113 MBytes   952 Mbits/sec                                                                                                                                                                                                                                                                                     
[  3]  2.0- 3.0 sec   114 MBytes   953 Mbits/sec                                                                                                                                                                                                                                                                                     
[  3]  3.0- 4.0 sec   114 MBytes   954 Mbits/sec                                                                                                                                                                                                                                                                                     
[  3]  4.0- 5.0 sec   114 MBytes   953 Mbits/sec                                                                                                                                                                                                                                                                                     
^[[B[  3]  5.0- 6.0 sec   114 MBytes   953 Mbits/sec                                                                                                                                                                                                                                                                                 
[  3]  6.0- 7.0 sec   114 MBytes   954 Mbits/sec                                                                                                                                                                                                                                                                                     
[  3]  7.0- 8.0 sec   114 MBytes   954 Mbits/sec                                                                                                                                                                                                                                                                                     
[  3]  8.0- 9.0 sec   114 MBytes   953 Mbits/sec
[  3]  9.0-10.0 sec   114 MBytes   953 Mbits/sec
[  3]  0.0-10.0 sec  1.11 GBytes   953 Mbits/sec
[  3] Sent 810691 datagrams

2. for multicast

 3]  0.0- 1.0 sec  11.7 MBytes  98.1 Mbits/sec
[  3]  1.0- 2.0 sec  11.4 MBytes  95.7 Mbits/sec
[  3]  2.0- 3.0 sec  11.4 MBytes  95.9 Mbits/sec
[  3]  3.0- 4.0 sec  11.4 MBytes  95.7 Mbits/sec
[  3]  4.0- 5.0 sec  11.4 MBytes  95.7 Mbits/sec
[  3]  5.0- 6.0 sec  11.4 MBytes  95.5 Mbits/sec
[  3]  6.0- 7.0 sec  11.4 MBytes  95.7 Mbits/sec
[  3]  7.0- 8.0 sec  11.4 MBytes  95.7 Mbits/sec
[  3]  8.0- 9.0 sec  11.4 MBytes  95.3 Mbits/sec
[  3]  9.0-10.0 sec  11.4 MBytes  95.7 Mbits/sec
[  3]  0.0-10.0 sec   114 MBytes  95.9 Mbits/sec

 

3. I change "AllowMuticast" parameter to "sdpd" and all works. but I not really understand what I did and I work with only 3 subscribers and don't really sure if we can work with >10 subscribers

Share this post


Link to post
Share on other sites

.. and for completeness .. here's a screenshot of a multicast 'run' between 3 machines (publisher on 'perf1', and 2 subscribers on 'perf2' and 'perf3'):

image.png

I've also included the 'top' output of the sending machine.

So here you can see that when sending 4K samples, a multicast throughput of appr. 850 mbps is achieved with a sending-side DDSI-transmit-thread load of 61% of 1 core (and a publisher-application load of 25% of another core (just forget the azure/influx/modbus stuff which isn't related to this test :) )

Note that I've used 'top' in Irix mode (so it shows the load on individual cores) and 'threads-enabled' (so that you can see the induced load of each individual thread).

PS> I used the ospl_sp_1gbps.xml config-file as that's fine for running on 1gbps networks. And as I've used a separate multicast-network via the 'enp2s0' interfaces, I've replaced the 'auto' interface-choide in that xml-file by ${INTERFACE} so that I can control the interface to use by an environment-variable which I've set to that 'enp2s0' interface-name i.e. like this:

image.png

 

 

Share this post


Link to post
Share on other sites

W.r.t. your results and changing multicast-usage for spdp only:

Changing that AllowMulticast setting to spdp implies that you use multicast ONLY for discovery yet NOT for the actual data-flows towards discovered endpoints.

So what you're than basically doing is to send repeatedly (over unicast) to each endpoint, which on a fast machine (like you have) is probably fine for a limited number of 'reader-machines' but with increasing numbers, you'll see the throughput dropping because the writer's DDSI 'xmit.user' thread will eventually max-out the core that its running on.

So if you get acceptable performance, its an option to consider but I'd surely verify if the root-cause (which I think is 'your network') can't be resolved.

In our case (where we observe that same performance drop and where the iperf version I've used also shows the 'dropped-messages' on the server-side as 88%), there's an 'uplink' to a 'fire-box' that apparently 'chokes' on multicast-traffic and with that cripples the multicast-performance of the switch that hosts the 'performance-machines' .. and using a cheap yet 'dedicated' 8-ports gbit-switch (that has no uplinks) has resolved that issue .. perhaps at least worth a try :)

Good luck,

-Hans

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...