BS in Deutsche Bank job ads regarding TCP/IP network latency (HFT)?

[spy]

But how is it used/applied in practice for HFT between a client/bot and the exchange/broker?
Does the exchange/broker side need to use such bypass-technique too, or does it use just the standard TCP/IP stack and it suffices to use such user-space TCP/IP on the client side only?
Or: is that intended only for own market making center, ie. for own internal/unofficial/non-public exchange or dark-pool?
And: which US exchanges support/offer this technique? Any links for official/corporate info?

Do yourself a favor @earth_imperator... polish up your résumé and take the first halfway decent job you can find at an exchange, investment bank, broker/dealer, or big hedge fund. Then come back to us in five or ten years and tell us how interesting and sexy HFT is.

You know who Brigitte Bardot is, right?

 

[Databento]

But how is it used/applied in practice for HFT between a client/bot and the exchange/broker?

On most venues, order messages are sent over a TCP session while data messages are sent over UDP multicast.

The trading participant can send or receive these messages through the specialized NIC and its userspace networking stack. Besides software optimizations, this networking stack takes advantage of any hardware offload capability of the NIC, e.g. TCP/UDP/IP checksum offload.

There are several ways that trading participants can integrate this networking stack. The most common way is to do nothing special with your trading application — just implement it with standard BSD sockets. Then you can invoke a proprietary application that comes with the NIC, which starts your trading application, and uses `LD_PRELOAD` to overwrite any standard Linux networking syscalls, steering those to the userspace network stack. Insodoing, all incoming data packets and outgoing order packets bypass the kernel are processed faster.

Does the exchange/broker side need to use such bypass-technique too,

Yes, some exchanges use this technique for their matching engine implementation.

Broker-dealers like Deutsche might use for any of their electronic trading activities, e.g. their algorithmic execution desk

And: which US exchanges support/offer this technique?

This is purely on client side so anyone can do it on any trading venue, so long as they're executing through a physical server.

Any links for official/corporate info?

Almost half of our engineering team has a HFT background.

 

[spy]

Almost half of our engineering team has a HFT background.

That's a great marketing blurb but let's face it... since:

"there is no single definition of HFT" (1)
​

basically anyone who trades and spent a few hours optimizing their software for speed of execution could make the same claim.

The term "HFT" is one of the most used and abused in the Wall St. zeitgeist I've ever seen; hollow and almost meaningless in today's age where everyone carries a supercomputer in their pocket.

 

[earth_imperator]

@Databento, thanks for the useful info.
Which Linux library for this can you recommend?
I have experience with low-level TCP/IP programming incl. raw sockets. but it was about a decade ago...

 

[earth_imperator]

Do yourself a favor @earth_imperator... polish up your résumé and take the first halfway decent job you can find at an exchange, investment bank, broker/dealer, or big hedge fund. Then come back to us in five or ten years and tell us how interesting and sexy HFT is.

BB I of course know

I indeed was interested at DB, asked their HR dept to give me a contact for some basic initial Qs by me to them. They said they have forwarded my request to their "Fachabteilung" and will inform me as soon as their dept replies. This was about 2 weeks ago... Still waiting for an answer from this Deutsche Bank in Berlin...
But I'm not the slightest surprised...
The job ads could be unreal, just marketing ads....

 

[earth_imperator]

Did some online research:

https://en.wikipedia.org/wiki/Express_Data_Path
"
XDP (eXpress Data Path) is an eBPF-based high-performance data path used to send and receive network packets at high rates by bypassing most of the operating system networking stack. It is merged in the Linux kernel since version 4.8.
...
The idea behind XDP is to add an early hook in the RX path of the kernel, and let a user supplied eBPF program decide the fate of the packet. The hook is placed in the network interface controller (NIC) driver just after the interrupt processing, and before any memory allocation needed by the network stack itself, because memory allocation can be an expensive operation. Due to this design, XDP can drop 26 million packets per second per core with commodity hardware.[4]
...
Along with XDP, a new address family entered in the Linux kernel starting 4.18. [9] AF_XDP, formerly known as AF_PACKETv4 (which was never included in the mainline kernel), [10] is a raw socket optimized for high performance packet processing and allows zero-copy between kernel and applications. As the socket can be used for both receiving and transmitting, it supports high performance network applications purely in user space. [11]
..."

See also DPDK:
https://en.wikipedia.org/wiki/Data_Plane_Development_Kit

 

[Databento]

That's a great marketing blurb but let's face it... since:

"there is no single definition of HFT" (1)
​

basically anyone who trades and spent a few hours optimizing their software for speed of execution could make the same claim.

The term "HFT" is one of the most used and abused in the Wall St. zeitgeist I've ever seen; hollow and almost meaningless in today's age where everyone carries a supercomputer in their pocket.

We mean it in the conventional sense: firms doing at least 2+% ADV in any given market. Roughly the top 30 firms in the US/Europe by trade frequency.

 

[Sprout]

On most venues, order messages are sent over a TCP session while data messages are sent over UDP multicast.

The trading participant can send or receive these messages through the specialized NIC and its userspace networking stack. Besides software optimizations, this networking stack takes advantage of any hardware offload capability of the NIC, e.g. TCP/UDP/IP checksum offload.

There are several ways that trading participants can integrate this networking stack. The most common way is to do nothing special with your trading application — just implement it with standard BSD sockets. Then you can invoke a proprietary application that comes with the NIC, which starts your trading application, and uses `LD_PRELOAD` to overwrite any standard Linux networking syscalls, steering those to the userspace network stack. Insodoing, all incoming data packets and outgoing order packets bypass the kernel are processed faster.



Yes, some exchanges use this technique for their matching engine implementation.

Broker-dealers like Deutsche might use for any of their electronic trading activities, e.g. their algorithmic execution desk



This is purely on client side so anyone can do it on any trading venue, so long as they're executing through a physical server.



Almost half of our engineering team has a HFT background.

How does SIP relate to what you just described?

 

[Sprout]

I won't deny there's a great deal of marketing/bull when these guys look to hire. However, bypassing network protocols such as TCP/IP can indeed improve latency.

They're most likely referring to something like [remote] direct memory access. Infiniband is one such technology that uses this method. And, if you want to do fast order matching in a clustered compute environment for a big dark pool or at the exchange itself... it would likely pay to skip the protocols.

Basically, you're linking up a bunch of computers on a memory bus/backplane so they'll share the same memory space.

Is this what SIP is built upon?

https://focus.world-exchanges.org/articles/nasdaq-market-data

 

[Sprout]

Btw, for the discriminating S&M practitioner:

OpenFabrics Alliance.
Oracle's take.
EPAM's pitch.
An academic paper.
State machine replication for high availability using RDMA.
Another pitch, this time from Arista networks.
Yada, yada, yada...

Package: librdmacm-dev          
Version: 22.1-1
State: not installed
Multi-Arch: same
Priority: optional
Section: libdevel
Maintainer: Benjamin Drung <benjamin.drung@cloud.ionos.com>
Architecture: amd64
Uncompressed Size: 325 k
Depends: libibverbs-dev, librdmacm1 (= 22.1-1)
Description: Development files for the librdmacm library
librdmacm is a library that allows applications to set up reliable connected and unreliable datagram
transfers when using RDMA adapters. It provides a transport-neutral interface in the sense that the same code
can be used for both InfiniBand and iWARP adapters.  The interface is based on sockets, but adapted for queue
pair (QP) based semantics: communication must use a specific RDMA device, and data transfers are
message-based.
librdmacm only provides communication management (connection setup and tear-down) and works in conjunction
with the verbs interface provided by libibverbs, which provides the interface used to actually transfer data.
This package is needed to compile programs against librdmacm1. It contains the header files and static
libraries (optionally) needed for compiling.
Homepage: https://github.com/linux-rdma/rdma-core
Tags: devel::library, role::devel-lib

Package: libibverbs-dev
Version: 22.1-1
State: not installed
Multi-Arch: same
Priority: optional
Section: libdevel
Maintainer: Benjamin Drung <benjamin.drung@cloud.ionos.com>
Architecture: amd64
Uncompressed Size: 1,397 k
Depends: ibverbs-providers (= 22.1-1), libibverbs1 (= 22.1-1), libnl-3-dev, libnl-route-3-dev
Description: Development files for the libibverbs library
libibverbs is a library that allows userspace processes to use RDMA "verbs" as described in the InfiniBand
Architecture Specification and the RDMA Protocol Verbs Specification.  iWARP ethernet NICs support RDMA over
hardware-offloaded TCP/IP, while InfiniBand is a high-throughput, low-latency networking technology.
InfiniBand host channel adapters (HCAs) and iWARP NICs commonly support direct hardware access from userspace
(kernel bypass), and libibverbs supports this when available.
This package is needed to compile programs against libibverbs1. It contains the header files and static
libraries (optionally) needed for compiling.
Homepage: https://github.com/linux-rdma/rdma-core
Tags: devel::library, role::devel-lib

Thank you for the links. What is "S&M Practitioner?"