Gemini WebSocket Terminal
Overview:
This project is a command-line WebSocket application that connects to the Gemini exchange to receive real-time market data feeds.
This code assumes usage of Gemini market data V2 for WebSockets. Endpoint:
host: api.gemini.com
port: 443
target: /v2/marketdata
The app first connects to a websocket, then spins up a writer/producer handler thread for that connection, while the main thread
acts as a reader/consumer. It also has support for real-time commands so the user can either change connection, close the app or
set order book depth.
At the moment of writing, the order book implementation uses 2 balanced maps to track bids and asks, with a hash map lookup cache
for faster order removals. To sync the producer and consumer I use 3 different methods: mutex, atomic load/store and wait-free queue.
These can be swapped around from within the 'Include.h' file. By ddefault it is set to use wait-free.
=============================
Features (Planned / WIP)
- Connect to Gemini WebSocket API
- Subscribe to order books of market data feeds (proposed BTCUSD, ideally any)
- Command-based terminal interface
- Real-time display of incoming messages
- Option to log messages
=============================
Technology Stack
`C++17`
- [macOS 26.2, Darwin Kernel 25.2.0]: I am using a mac mini with M4 CPU for this project
- [Boost 1.90]: Used for general utilities
- [OpenSSL 3.6.2]: Using OpenSSL for TLS resolution
- [nlohmann 3.12.0]: Json parser
I decided to port the project on windows WSL (Ubuntu 24.04.3 LTS) too. To make it work I had to do a bunch of global installs:
sudo apt install libboost-all-dev [https://www.developnsolve.com/linux/how-to-install-boost-linux] (to install boost)
sudo apt install libssl-dev [https://www.developnsolve.com/linux/how-to-install-openssl-in-linux] (to install openssl)
sudo apt install build-essential [https://www.geeksforgeeks.org/installation-guide/how-to-install-gcc-compiler-on-linux/] (to install g++)
sudo apt install nlohmann-json3-dev [https://ubuntu.pkgs.org/24.04/ubuntu-universe-amd64/nlohmann-json3-dev_3.11.3-1_all.deb.html] (to install the json parser)
sudo apt install libncurses5-dev [https://www.developnsolve.com/linux/how-to-install-ncurses-linux] (to install ncurses)
What has been attempted, but ended up discarded
- [WebSocket++ 0.8.2](https://github.com/zaphoyd/websocketpp): WebSocket client library
=============================
Setup
1. Clone this repository:
```bash
git clone http://10.0.0.2:9121/alymanie/WebSocket-Cmd-Fin.git websocket_project
cd websocket_project
```
2. Build the project
```bash
./build.sh
```
3. Use helper arguments when running to get a grasp
```bash
./run.sh -h
```
4. Connecting to Gemini market data endpoint for BTCUSD
```bash
./run.sh -c api.gemini.com 443 /v2/marketdata BTCUSD
```
5. Multi instrument support
```bash
./run.sh -c api.gemini.com 443 /v2/marketdata BTCUSD,ETHUSD,SOLUSD,XRPUSD,DOGEUSD,DOTUSD
```
=============================
Documentation used:
1. https://docs.websocketpp.org/md_tutorials_utility_client_utility_client.html
2. https://tests.ws/guides/cpp-websocket
3. https://www.boost.org/doc/libs/1_85_0/libs/mysql/doc/html/mysql/examples/ssl.html
4. https://www.boost.org/doc/libs/latest/libs/beast/doc/html/beast/using_io/ssl_tls_certificate.html
5. https://www.boost.org/doc/libs/latest/libs/beast/doc/html/beast/examples.html [Extremely useful examples]
6. https://www.boost.org/doc/libs/latest/libs/beast/example/websocket/client/sync-ssl/websocket_client_sync_ssl.cpp
7. https://stackoverflow.com/questions/72795727/boost-asio-ssl-handshake-failure
8. https://www.boost.org/doc/libs/latest/doc/html/boost_asio/reference/ssl__context/set_default_verify_paths/overload1.html
9. https://hackaday.com/2025/06/17/a-gentle-introduction-to-ncurses-for-the-terminally-impatient/
10. https://github.com/milan-r-shah/CppND-System-Monitor-Project-Updated
11. https://www.youtube.com/watch?v=g7Woz3YVgvQ
12. https://www.youtube.com/watch?v=2Q0jUzhDJp0
13. https://www.youtube.com/watch?v=vfNI0n5vFLE&list=PL2U2TQ__OrQ8jTf0_noNKtHMuYlyxQl4v&index=7 [https://www.youtube.com/@CasualCoder This channel saved the day. It has numerous tutorials for many use cases for ncureses.h]
14. https://stackoverflow.com/questions/1641182/how-can-i-catch-a-ctrl-c-event
15. https://www.studyplan.dev/pro-cpp/json
16. https://stackoverflow.com/questions/55431552/how-to-iterate-over-a-json-in-json-for-modern-c
17. https://www.reddit.com/r/C_Programming/comments/b6gv0g/ncurses_window_resize_trigger/
18. https://www.youtube.com/watch?v=sX2nF1fW7kI
19. https://runebook.dev/en/docs/cpp/memory/shared_ptr/atomic2
20. https://stackoverflow.com/questions/70440813/correct-usage-of-stdatomicstdshared-ptrt-with-non-trivial-object
21. https://stackoverflow.com/questions/62923348/state-of-stdatomic-shared-ptr [which opens up the c++ docs https://en.cppreference.com/w/cpp/memory/shared_ptr/atomic]
22. https://wryzxec.github.io/lockfree_spsc.html
23. https://github.com/wryzxec/PikaQ/blob/main/include/pika_queue.hpp
24. https://stackoverflow.com/questions/152016/detecting-cpu-architecture-compile-time
25. https://tonygo.tech/blog/2023/define-cpu-architecture-in-cpp
26. https://trofi.github.io/posts/233-ncurses-update-journey.html
27. https://stackoverflow.com/questions/21046742/using-boostsystemerror-code-in-c
28. https://stackoverflow.com/questions/79587242/boost-beast-websocket-ssl-operation-cancelled-error-when-attempting-controlle [this shows how to properly shutdown a socket]
29. https://stackoverflow.com/questions/60997939/what-exacty-is-io-context
30. https://www.boost.org/doc/libs/1_82_0/doc/html/boost_asio/overview/basics.html
=============================
Difficulties:
1. I hit a wall afterfinally getting into actually implementing RAII for WebSockets
endpoints using websocket++ lib. Turns out, on a MacOS running M4 it is not trivial
to obtain access to a functional older boost version < 1.70.
This is an issue because websocket++ latest uses an older < 1.70 version for boost,
which is not compatible on an apple silicon CPU, which is what I am using for this project.
I will attempt next to use a low-er level solution... going straight to Boost.Beast
2. I have a starting point, I can reach the gemini endpoint via websockets, but the handshake gets decliened:
The WebSocket connection failed. ['./bin/app -c api.gemini.com 443 /v1/marketdata/BTCUSD'] Reason: Exception while attempting to connect to [api.gemini.com:443]: [The WebSocket handshake was declined by the remote peer [boost.beast.websocket:20 at /opt/homebrew/opt/boost/include/boost/beast/websocket/impl/stream_impl.hpp:657:40 in function 'operator()']]
That checks out... after all the endpoint uses TLS while I attempt raw websockets... I assume claudflare intervienes there. Time to change my implementation to adapt ssl
3. Next hurdle: The WebSocket connection failed. ['./bin/app -c api.gemini.com 443 /v1/marketdata/BTCUSD'] Reason: Exception while attempting to connect to [api.gemini.com:443]: [handshake: certificate verify failed (SSL routines) [asio.ssl:167772294]]
Somehow I need to fix the certificate verify for the handshake. Time to read more docs.
Found a potential solution on https://stackoverflow.com/questions/72795727/boost-asio-ssl-handshake-failure
Particularly, the forum post on stack overflow seems to use the following "ssl_context.set_default_verify_paths();". The docs seem to mention it is used to specify the system certs:
https://www.boost.org/doc/libs/latest/doc/html/boost_asio/reference/ssl__context/set_default_verify_paths/overload1.html
4. Now that WebSocket TLS connection to Gemini works, I would like a bit of a better terminal display. Time to learn some basic 'ncurses.h'.
These youtube tutorials helped a lot "https://www.youtube.com/watch?v=g7Woz3YVgvQ" "https://www.youtube.com/watch?v=2Q0jUzhDJp0"
5. Time to create the order book structure and logic. Here is a great video about key tradeoffs: https://www.youtube.com/watch?v=sX2nF1fW7kI
6. Now the difficult part... since I made this app allow multi-subscription to instruments under the same venue/exchange, I now am stuck at how to write to each order book one one thread and read
from the same books from another (main) thread. I could do mutexes on all book calls, that is the simple way out. But having intrerupt sys calls on a critical path is not ideal.
The solution would be to achieve lock-free queue between my two threads somehow, and since I have not been in this situation before, it's time I read a little more docs.
I do, however, believe, it is possible with std::atomic to store the book that is currently being updated in the writer thread and then from the reader thread I attempt to load each book anyway to
display, but i need to make sure this works. Technically it should work, since the app avoids contention on writes by design (well, except the logger, but that's a different subject).
Ah, since I am at this stage, I will have to consider thread affinity... allocating the worker/writer thread to a particular cpu core. Idea is, trying to minimise cache line ping pong, and avoid the scheduler.
I found a cool example of this in practice: https://runebook.dev/en/docs/cpp/memory/shared_ptr/atomic2
Aaaand I was hit with the following: error: _Atomic cannot be applied to type 'std::shared_ptr<OrderBook>' which is not trivially copyable
Turns out I cannot just simply wrapt a shared_ptr into an atomic... who would have thought?
I found this post on stack overflow with a solution https://stackoverflow.com/questions/62923348/state-of-stdatomic-shared-ptr Just use std::atomic_ wrappers!!!
7. Now i got cross book entries... I need to debug
Ok... found out a potential cause. Because I am not yet deleting price entries at all, if the prices go up on both BID and ASK, and then down, because I am not
removing, the higher BIDs from the previious up wave are still present while the AKS, being in ascending order, will go down in value, resulting in a cross...
I really really need to figure out how this venue handles order book entry deletes....
under the provided docs at https://docs.gemini.com/websocket/archived/order-events
I found this line "When limit orders are booked, they have a non-zero quantity visible on the exchange. Market orders are never booked"
This makes me wonder what happens when I get a l2 update with quantity = 0
Ahh finally a winner: "Quantity zero indicates price level removal." on https://docs.gemini.com/websocket/streams#balance-updates
This might not be the same case for me, but I will give it a shot.
8. Now I have a new bigger issue... runtime seg fault somewhere... will probably be a nightmare to debug.
My theory now is that the .reserve() I do on the unordered_map holding all order books might be a bad bad ide. Primarily
because it's hard for me to track the impact the ABI implementation might have in a multi-threaded architecture.
I will change that to be reserved from its instantiation instead.
This will be fun once I implement the option to swap to another market data provider... (different connectivity and all)
I also deciede to instantiate the cache before I start processing any of the threads
I also realized I was not making a copy of the atomic load result in the writer.... And this releaved the azctual problem.
My OrderBook implementation is non-copiable because of the iterators used in it
I think I will discard the atomic approach and switch back to mutexes.
9. Ok...ok... Took me a little longer than expected... but I just realized this venue obviously also sends heartbeats...
So far I considered it would somehow magically ony send book updates... I have to rethink a few things in the design...
That was the main cause for why the atomic load/store logic would cause secfaults at times... now it works
10. Now I am working on displaying metrics and improving critical path performance. I found the following wait-free queue implementation:
https://wryzxec.github.io/lockfree_spsc.html
I am planning to change the current architecture to enqueue book updates on writer and dequeue (which would display and populate the book cache)
on the reader. This will work fairly nice mainly because the reader is much faster than the writer + I only have 1 consumer and 1 producer.
I am also planning to change the OrderBook.h implementation and move to using vectors instead of maps, and defining copy constructor to handle
copying iterators properly for the hash map lookup
11. After attempting to port the source code to ubuntu, compilation shows the follwoing warnings:
src/Commands.cpp:513:18: warning: format not a string literal and no format arguments [-Wformat-security]
513 | mvwprintw(mOrderBookDisplay, 0, (mDisplayData.wOrdBook - topTitle.size()) * 0.5, topTitle.c_str());
| ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/Commands.cpp:558:30: warning: format not a string literal and no format arguments [-Wformat-security]
558 | mvwprintw(mOrderBookDisplay, verticalOffset + verticalOffset * (mDisplayData.orderBookDepth + 1) + i + 1, bookAlign + 1, bookLine.c_str());
| ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/Commands.cpp:579:18: warning: format not a string literal and no format arguments [-Wformat-security]
579 | mvwprintw(mOutputDisplay, 0, 1, stats.c_str());
| ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/Commands.cpp:580:18: warning: format not a string literal and no format arguments [-Wformat-security]
580 | mvwprintw(mOutputDisplay, 1, 1, debug.c_str());
| ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/Commands.cpp:581:35: warning: format not a string literal and no format arguments [-Wformat-security]
581 | if(output != "") mvwprintw(mOutputDisplay, 3, 1, output.c_str());
After a bit of google-ing, I found this explanation on it: https://trofi.github.io/posts/233-ncurses-update-journey.html
Basically, 'mvprintw' can cause arbitrary memory access. which should have been obvious from the get-go...
Seems it escaped me to keep consistent with the formatting there on all calls to 'mvwprintw'
12. I am currently trying to implement websocket failover handling:
2026-Apr-16 18:02:14.510260: [INFO]: Successfully connected to [api.gemini.com:443/v2/marketdata|BTCUSD] on connection id [1]
2026-Apr-16 18:02:27.448605: [ERROR]: Failure to read WebSocket ssl buffer. Unhandled error state... [104: Connection reset by peer]
2026-Apr-16 18:02:27.943329: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [125: Operation canceled]
2026-Apr-16 18:02:27.943536: [ERROR]: Cannot read from connection id [1] because that id does not point to a valid connection. ['recv']
2026-Apr-16 18:02:27.943682: [ERROR]: Cannot read from connection id [1] because that id does not point to a valid connection. ['recv']
To test I am using an ubuntu linux distro only for the ability to simulate a connection drop using the following commands:
alymanie@UgaBunga:/mnt/c/Coding/c++/WebSocket-Cmd-Fin$ netstat -antp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.54:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -
tcp 0 0 10.255.255.254:53 0.0.0.0:* LISTEN -
tcp 0 0 xxx.xxx.xxx.xxx:40272 52.1.164.60:443 ESTABLISHED 4948/./bin/app
alymanie@UgaBunga:/mnt/c/Coding/c++/WebSocket-Cmd-Fin$ sudo tcpkill host 52.1.164.60 and port 443
tcpkill: listening on eth0 [host 52.1.164.60 and port 443]
52.1.164.60:443 > xxx.xxx.xxx.xxx:40272: R xxxxxxxxx:xxxxxxxxx(0) win 0
52.1.164.60:443 > xxx.xxx.xxx.xxx:40272: R xxxxxxxxx:xxxxxxxxx(0) win 0
52.1.164.60:443 > xxx.xxx.xxx.xxx:40272: R xxxxxxxxx:xxxxxxxxx(0) win 0
After tweaking and retesting, now I get a seg fault on reconnect:
BID BTCUSD ASK
│ 75033.01000 0.210518|75033.02000 0.082000
│ 75032.47000 0.013979|75033.53000 0.025300
│ 75028.00000 15.600000|75033.54000 0.348221
│ 75027.30000 0.066630|75036.94000 0.034985
│ 75027.00000 0.300000|75038.88000 0.066631
│ ./run.sh: line 1: 4948 Segmentation fault (core dumped) ./bin/app "$@"
I got some more progress:
2026-Apr-16 20:03:29.108282: [INFO]: Successfully connected to [api.gemini.com:443/v2/marketdata|BTCUSD] on connection id [1]
2026-Apr-16 20:03:45.368183: [ERROR]: Failure to read WebSocket ssl buffer. Unhandled error state... [104: Connection reset by peer]
2026-Apr-16 20:03:45.898215: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [125: Operation canceled]
2026-Apr-16 20:03:45.898390: [WARN]: Attempting reconnect...
2026-Apr-16 20:10:30.920187: [INFO]: Successfully connected to [api.gemini.com:443/v2/marketdata|BTCUSD] on connection id [1]
2026-Apr-16 20:10:46.872603: [ERROR]: Failure to read WebSocket ssl buffer. Unhandled error state... [104: Connection reset by peer]
2026-Apr-16 20:10:47.407307: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [125: Operation canceled]
2026-Apr-16 20:10:47.407474: [WARN]: Attempting reconnect...
but still a seg fault...
┌────────────────────────────────────────────────────────────────Live data──────────────────────────────────────────────
│ BID BTCUSD ASK
│ 74856.17000 0.110474|74856.18000 0.196941
│ 74856.16000 0.014031|74864.09000 0.048869
│ 74847.41000 0.020000|74868.95000 0.340234
│ 74846.41000 0.029382|74869.52000 0.026700
│ 74846.10000 0.300000|74873.27000 0.004000
│
│
│
│
│
│
│
│
│
│
│
│
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
STATS-> WMMS: 18 WAMS: 3 RMS:0.93 WMS: 0.10 MPS: 163 MSGS: 2447
DEBUG-> BTCUSD:1:2630:2747 ./run.sh: line 1: 5005 Segmentation fault (core dumped) ./bin/app "$@"
Turns out my recv() function is in a bad state after reconnection... makes me sceptical of my way of handling the io_context:
https://stackoverflow.com/questions/60997939/what-exacty-is-io-context
After fixing that (duplicate entries, acessing deleted memory segments...) and also making sure I do not increase the shared_ptr
ref count when I try to get access to the connection data.... now I made it work.
Some final... progress?:
2026-Apr-17 17:03:34.433154: [INFO]: Successfully connected to [api.gemini.com:443/v2/marketdata|EURUSD] on connection id [1]
2026-Apr-17 17:03:57.361563: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [104: Connection reset by peer]
Successfully reconnected to [api.gemini.com:443/v2/marketdata|EURUSD] on connection id [1]
2026-Apr-17 17:04:02.813167: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [104: Connection reset by peer]
Exception while attempting to connect to [api.gemini.com:443] [/v2/marketdata]: [Connection reset by peer [system:104]]
2026-Apr-17 17:04:02.813384: [ERROR]: Failed to send subscription msg: [Cannot send [{"subscriptions":[{"name":"l2","symbols":["EURUSD"],"top_of_book":false}],"type":"subscribe"}] to connection id [1] because that id does not point to a valid connection.]
2026-Apr-17 17:04:07.813850: [ERROR]: Cannot read from connection id [1] because that id does not point to a valid connection. ['recv']
2026-Apr-17 17:04:07.814105: [WARN]: Disconnected. Attempting reconnect...
2026-Apr-17 17:04:07.814283: [ERROR]: Cannot read from connection id [1] because that id does not point to a valid connection. ['reconnect]
It does reconnect and resub, but sometimes, if timings are tight, the connection gets dropped during 'connect()' phase, dropping the connection permanently
After many attempts and tests I finally managed to implement websocket connectivity failover for the following 3 cases:
1) simple, direct connection pipe drop:
2026-Apr-17 22:42:57.895685: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [104: Connection reset by peer]
Successfully reconnected to [api.gemini.com:443/v2/marketdata|BTCUSD] on connection id [1]
2) indirect connectivity pipe failed and reported by the 'sent' msg:
2026-Apr-17 22:49:17.579556: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [104: Connection reset by peer]
Successfully reconnected to [api.gemini.com:443/v2/marketdata|BTCUSD] on connection id [1]
2026-Apr-17 22:49:18.580272: [ERROR]: Failed to send subscription msg: [Exception while attempting to send [{"subscriptions":[{"name":"l2","symbols":["BTCUSD"],"top_of_book":false}],"type":"subscribe"}] to connection id [1]: [Connection reset by peer [system:104]]]
2026-Apr-17 22:49:23.580696: [ERROR]: Connection [1] is recovering from an outage...
2026-Apr-17 22:49:23.580933: [WARN]: Disconnected. Attempting reconnect...
3) indirect connectivity pipe failure on attempting reconnect durin cal lto 'connect':
2026-Apr-17 22:51:14.611681: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [104: Connection reset by peer]
Successfully reconnected to [api.gemini.com:443/v2/marketdata|BTCUSD] on connection id [1]
2026-Apr-17 22:51:36.889439: [ERROR]: Failure to read WebSocket ssl buffer. Connection timeout... [104: Connection reset by peer]
Exception while attempting to connect to [api.gemini.com:443] [/v2/marketdata]: [handshake: Connection reset by peer [system:104]]
From what I tested, gemini endpoint domain name resolves to many different ips, so when I kill the currently running connectivity pipe, the recovery will re-attempt to resolve the host.
Because of this, sometimes, the resolved real ip can be the same as the one I am currently dropping from, this causes the 3rd symptom described above.
Other times, the reconnectivity is successful, but when attempting to send a message, it may be that the host ended up reseting the connection anyway and the app simply received late
a termination with reason "reset by peer", I believe that is what is causing the 2nd symptom. The first symptom is straight forward proper recovery
Notes:
I added mutex vs atomic load/store vs wait-free queue methods of dealing with the critical path for personal learning purposes
I observed reported times in the 'Stats' section for speeds when subscribing to all 6 of the following simultaneously: BTCUSD,ETHUSD,SOLUSD,XRPUSD,DOGEUSD,DOTUSD
With wait-free queue I notice sub 0.12 ms producer time
With atomic load/store (which does a copy of the order book on each update + not using the iterators hash map), i see times close to 1 ms producer time
With guard_lock mutex, I notice up to 0.400 ms producer time
I have not yet used perf before, but I will try to set it up for better performance analysis.
I added an LRU cache at one point. Its purpose was to manage different marketdata connection endpoints, not only gemini. But it is currently untested not finished,
the app does not support multi-venue connections correctly just yet.
I would also like to add the following in the future:
1) A spinlock and lock-free queue to benchmark with
2) A vector-based OrderBook implementation, which theoretically should improve CPU cache locality
3) Add command to add/remove subscriptions in real-time
4) Perf
5) Support for multiple connections
This project is a self-thought/learning coursework that required heavy research on top of personal knowledge and experience in the domain.
There still are many areas where my knowledge is lacking in experience/exposure, such as proper performance analysis,
ncurses display/UI knowledge, proper multithreaded systems/architecture, etc.
The architecture that I went with for this project makes use of the single producer single consumer logic, due to its simplicity and
effectiveness.
The end goal is to have a comand line-like websocket client that acts as a feed handler for order books, and ideally, can support hundreds of subscriptions.