Why is this UDP unicast not being received by the client?

[i960]

message protocol used by cme https://github.com/real-logic/simple-binary-encoding

The lack of a pure C implementation and generally non-POSIX like feel of this repo says everything about how robust I'd expect it to be.

Last I checked on this simply finding a formal spec for the protocol and server side behavior was a hassle (when it most certainly shouldn't be) and this is typical of software and standards involved with the financial industry.

 

[dumpdapump]

Somewhat disagree: while software has until recently often played a subordinate role in finance and management did not categorize IT within banks or hedge funds as an essential function (there are exceptions), nowadays exchange feeds and messaging, either FIX or exchange APIs are very robust, well defined, and well understood. It would lead to terrible mistakes if that was not the case. And remember, most financial firms hardly ever reinvent nor start over on the IT side: they stack technology on top of legacy technology that cannot be removed because it must support existing consumers. Deutsche Bank I heard has over 200,000 different software applications and sometimes dozens, if not hundreds of software stacks to address an identical issue.

The lack of a pure C implementation and generally non-POSIX like feel of this repo says everything about how robust I'd expect it to be.

Last I checked on this simply finding a formal spec for the protocol and server side behavior was a hassle (when it most certainly shouldn't be) and this is typical of software and standards involved with the financial industry.

 

[kmiklas]

May I inquire what you are actually trying to accomplish with this entire project?

We are trying to accomplish the creation of an exchange; very much similar to how Josh Levine created Island. We are building an entitiy to compete with the Nyse, Nasdaq, et. al, that it is built upon Free Software Foundation (FSF) and open-source principles: free market data, open DOM, etc. The concept is to remove positions of privilege (think $100M+/year colo fees, $6K/month non-display fees, etc) and level the playing field. All traders, speculators, investors, and gamblers alike will have access to the same information at the same price. At heart, Ritchie is fundamentally aligned with the SEC mission (emphasis mine):

The mission of the U.S. Securities and Exchange Commission is to protect investors, maintain fair, orderly, and efficient markets, and facilitate capital formation.

https://www.sec.gov/Article/whatwedo.html

Note that, although market data are free, trades are not. The exchange shall charge fees, and broker/dealers will charge commission--equal and fair fees for all market participants.

You want to share market data? Where is that market data coming from?

That market data is coming from the Ritchie market data server, just like any other exchange. I am coding it right now, and a technical issue with UDP was the impetus for posting this thread. I'm now using sample prices, because the exchange code isn't finished yet, but soon prices will be based on the market value of assets traded on the exchange. We are working hard to get paper trading up and running.

As you describe it on your website it does not seem to be in compliance with any exchange or other data provider contract

Unless I misunderstand your meaning of "exchange" and "data provider," I believe that compliance with the SEC and FINRA are more relevant here. I have a relationship with a law firm in Washington. One of the "Keys to Success" in our business plan is for Ritchie to be fully compliant with all FINRA, SEC, and other applicable regulations. So important is this point, that one of our first IPOs will be an innovative service designed to empower the SEC to better monitor and enforce compliance.

I have been devoting my time to building the market data server, and haven't had a chance to update the Web site recently.

 

[kmiklas]

Your problem is this:

$ strace -f ./talker localhost foo
[..]
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1496375815, pid=14048}, "\0\0\0\0"}, {{len=1, type=0x14 /* NLMSG_??? */, flags=NLM_F_REQUEST, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20
close(3)                                = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(4950), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
getsockname(3, {sa_family=AF_INET, sin_port=htons(31436), sin_addr=inet_addr("127.0.0.1")}, [28->16]) = 0
close(3)                                = 0
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET6, sin6_port=htons(4950), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 0
getsockname(3, {sa_family=AF_INET6, sin6_port=htons(51297), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0
close(3)                                = 0
socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDP) = 3
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
write(1, "Price multicasted: 50.34\n", 25Price multicasted: 50.34) = 25
sendto(3, "50.34", 5, 0, {sa_family=AF_INET6, sin6_port=htons(4950), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 5
nanosleep({1, 0}, 0x7fff12a85070)       = 0
write(1, "Price multicasted: 50.36\n", 25Price multicasted: 50.36) = 25
sendto(3, "50.36", 5, 0, {sa_family=AF_INET6, sin6_port=htons(4950), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 5
nanosleep({1, 0}, 0x7fff12a85070)       = 0
write(1, "Price multicasted: 50.27\n", 25Price multicasted: 50.27) = 25
sendto(3, "50.27", 5, 0, {sa_family=AF_INET6, sin6_port=htons(4950), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 5
nanosleep({1, 0}, 0x7fff12a85070)       = 0
write(1, "Price multicasted: 49.65\n", 25Price multicasted: 49.65) = 25
sendto(3, "49.65", 5, 0, {sa_family=AF_INET6, sin6_port=htons(4950), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 5
nanosleep({1, 0}, ^Cstrace: Process 14048 detached
<detached ...>

This section of code looks for any interface to bind to and then bails after the first socket call that works. Problem is it doesn't care if the thing it finds is AF_INET6 and happily uses IPv6 to try and talk to the IPv4 listener on the other side (because you hard-specify AF_INET in listener.c):

   // loop through all the results and make a socket

    for(p = servinfo; p != NULL; p = p->ai_next) {
        if ((sockfd = socket(p->ai_family, p->ai_socktype,
                            p->ai_protocol)) == -1) {
            perror("talker: socket");
            continue;
        }

        break;
    }

Either add this in that loop:

       if(p->ai_family != AF_INET)
                continue;

Or get rid of that loop you probably don't even know why you're using. With that change it works "fine" for me:

$ ./talker localhost foo
Price multicasted: 50.34
Price multicasted: 50.36
Price multicasted: 50.27
Price multicasted: 49.65
Price multicasted: 50.43
Price multicasted: 49.86
Price multicasted: 50.36
Price multicasted: 50.43
Price multicasted: 50.00

$ ./listener
listener: waiting to recvfrom...
listener: got packet from 127.0.0.1
listener: packet is 5 bytes long
listener: packet contains "50.43"
listener: got packet from 127.0.0.1
listener: packet is 5 bytes long
listener: packet contains "49.86"
listener: got packet from 127.0.0.1
listener: packet is 5 bytes long
listener: packet contains "50.36"
listener: got packet from 127.0.0.1
listener: packet is 5 bytes long
listener: packet contains "50.43"

You're really outgunned here with what you think you're dealing with. This client/server code is some of the most basic "my first BSD sockets" program out there and this is why you're going to run into problems:

• Writing C++ code inside .c files and thinking it's C. It isn't C, it's some bastardized form of C with C++ mixed in. Stop doing that. I had to remove certain calls from your code just to get it to compile under gcc.
• Extremely basic socket handling that isn't even going to remotely scale. At the minimum an event-loop and an IO multiplexing framework is *required* (i.e. select, poll, epoll, libevent, etc). In addition everything *has* to be done in a non-blocking fashion within an asynchronous framework/paradigm. Threading at some point most likely also comes in - but NOT as a replacement for a robust client/server connection-handling framework.
• You're sending random things across the wire by converting floating point to a string and dumping it to the other side. This is trivial and nobody does it this way. It's not even remotely a usable or efficient protocol.
• You're using tcpdump to debug things when you should be using strace and ltrace w/ tcpdump as a final sanity check.
• Unless you've read Stevens UNPv1, and have AUPv1 and TCP/IP Illustrated Vol 1-2 on your shelf for reference you're completely shooting in the dark. You don't know what you don't know.
• I can only imagine what the memory handling code is going to look like. Hope you learn how to use valgrind at some point, because you're gonna need it.

I have 20+ years experience in this stuff and this wouldn't even make it past a code review. You're attempting to write your own exchange but have zero real world POSIX C network programming experience. I'm very supportive of people trying to learn new things but c'mon man you need to get way more fucking realistic about what you're getting into.

@i960 , tick data test server is up and running. Kindly hit the URL and let me know if you see data flowing:
http://www.ritchiestockexchange.com/tickdata/index.html

- The exchange, market data server, and Web data server are all separate
- "Last sale" price data are sent via C/C++ socket connection to the market data server.
- The market data server sends data via another socket connection to a Nodejs Websocket server
- The Nodejs establishes websocket connections to subscribers. Subscription happens automatically on page load.
- Planned is to enable UDP subscriptions directly to the market data server.

Also, please take a look at the site itself, and let me know if anything appeals to you in terms of contribution.
http://www.ritchiestockexchange.com/