> But if I had any kind of point, it would probably be that spending the time to do things like write tools and explore the debugging space is pretty much always worth it.
> Strangely, I've found that there is a not-insignificant number of people are against this - especially in a professional environment. I think this has something to do with the JIRA-fication of development process, where all work is divided and sub-divided, and anything that does not directly check a box and produce a deliverable is wasted effort. It's disheartening that there is a whole group of people - developers and managers alike - who do not understand or see the value in exploration as part of the process.
So true, being able to create your own small tools is a superpower which is often at the heart of 10x programmers. It is sadly an art often practiced in the shadow.
Since the title is rather vague, this is the start of a series about building a TCP/IP and Ethernet framing stack from scratch for a microcontroller. The author uses a chip (W5100) which can handle TCP/IP itself, but also supports handing it pre-built Ethernet frames, although the chip handles the preamble and CRC calculation. Most of the article is about trying to communicate with the chip itself, and sending a test packet (which I'm guessing is hardcoded, although it's not called out in the article).
(I was hoping it would be about bit-banging Ethernet on some improbable bit of hardware.)
I took a strange career jump recently into FPGA engineering with a focus on Ethernet. It's been a fun journey, which culminated in me finally designing my own Hard MAC IP and sending a packet over some custom PHY IP. I'd highly recommend it for those looking to try a "Hard Mode" version of this challenge. I feel like networking is very abstracted from users, so understanding how Ethernet cards, modems and switches put together and pull apart sections of packets, and how the PHY/PCS recovers signals over a link was really valuable.
First time seeing the retcons for MOSI/MISO: main out/subordinate in instead of master out/slave in. I'll use that so I can still refer to the pins as MOSI/MISO. The COPI/CIPO alternative (controller out/peripheral in) never worked themselves into my brain stem properly.
I'm baffled as to why the author is using STM32F401 with a W5100 Ethernet shield when they could just as easily use an STM32F407 board, which includes a built-in Ethernet MAC, coupled with a cheap (2 for ~$12) Ethernet PHY board. Lots of example Ethernet projects available and just as easy to develop for as the STM32F401.
Also,
> Due to the complexity of the signalling involved with ethernet, a dedicated ASIC is generally used
In the context of microcontrollers, I think this is generally not true. In most cases Ethernet functionality is incorporated as a embedded peripheral within the microcontroller itself, as it is with the STM32F407 and the ESP32 (both of which happen to use the same Ethernet peripheral sourced from an external provider).
If you've got a hankering for wired ethernet on a microcontroller, several of the larger STM32 Nucleo boards [1] have 100Mbps ethernet built in - and at ~$25 they're pretty affordable.
Their 'STM32Cube' software gets mixed reviews, but it will spit out a working example of ethernet communication.
note that if you want to play with writing your own network stack on linux, you can use `socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))` and operate on not too dissimilar level of abstraction as this is.
SOCK_RAW packets are passed to and from the device driver without
any changes in the packet data. When receiving a packet, the
address is still parsed and passed in a standard sockaddr_ll
address structure. When transmitting a packet, the user-supplied
buffer should contain the physical-layer header. That packet is
then queued unmodified to the network driver of the interface
defined by the destination address.
A short story in the data analysis space, a place where clever and underpaid people are discouraged from materializing logic into tables.
It took me a while to realize the data consumers (those who need to look at and understand data) didnt have any orchestration tools, except a creaky UI cron type of scheduler.
For my work I refused to compromise and spend all my days clicking in a UI, so I searched around for a good tool, decided on Prefect. It served me well, but I didn't realize going in that Prefect/Airflow/Argo etc are really orchestration engines and you still need to write the client software around it to serve your own (or the team) productivity.
For example, connections to sources, methods to extract from source with a sql file, where to put the data after, injecting into publication channels, etc. I all gradually ended up writing functions to support this for my own personal IC work. I sank a ton of time into learning, had to learn how to develop python packages (I chose poetry), on an on. And yet despite all that time spent grinding away on indirectly productive learning I have still been more productive than my peers.
It was a mystery so I spent time learning about my peer workflows, which people are so cagey about at my company. Anyway, everybody just crams 100s of lines of business logic into Tableau custom sql sources to avoid scrutiny of data artifacts like tables they might create on the data lake. I guess these are tableau flavored views, but its so hard to read and understand all this logic in the UI -- oh and the calculated fields also.
I guess to sum up, if I am keeping up with peers using the latest expensive enterprise 'data storytelling' service while I self educate on basically data engineering to stats presentation plots, spark, julia, python, etc. I think I have concluded that Tableau should be considered harmful.
As a hobbyist I wish that electronic gods would invent a way to bit-bang real world protocols. It is possible with simple protocols, but when we move to MHz range, it moves to impossible. Some kind of co-processor which would work at, say, 100 GHz speed with very limited instruction set. So I could implement some simple protocol parsing and buffering for more complex processing at lower speeds. Probably not possible, but who knows...
I used the W5100 before with Arduino but I am not sure about "gold standard" quality. I had a lot of issues with random disconnects several times a day with this board. Maybe it's the board, or maybe it's the arduino code or the connection. Or maybe it was my code. I don't know. Eventually I switched to a pi and never had any connection problems anymore.
When it comes to tcp connections linux seems more robust.
Nicely done! I just spent my last 16h of work time on implementing the reverse: parsing ethernet2 (with vlans!), ipv4+6 & UDP up to an automotive IP protocol. Use case is to understand a proprietary bus capturing stream that forwards ethernet frames (among other things) nested in ethernet frames that I receive on a raw socket. Fun stuff! Wireshark & Chatgpt are invaluable for this kind of task
Back in the late 90's, when there was very little hand holding, I build a little web server board using an 8051. It was fun to learn the internals of a TCP/IP stack and magical when it served its first web page. Dang was it slow though!
The moment I received my first packet on a cut-up wired headset I used as a transceiver makeshift tenceiver it felt like something clicked and I just began to understand more how the universe works. I recommend projects of this type and wish more folks did it.
Nice. I like those Ethernet chips. These days, you can find MCUs with built-in Ethernet hardware, but I still enjoy working with simple AVRs, so the w5500 (basically an upgraded w5100) is my go-to.
I wrote a little framework (bootloader, utilities, etc.) to allow for updating firmware and tunneling stdout on AVRs over pure Ethernet.
You may want to also consider ENC28J60 'Stand-Alone Ethernet Controller with SPI Interface' which lacks a hardware TCP/IP stack and is purely an Ethernet MAC and PHY accessible over SPI. This simplicity means it should be easier for you to get started with sending an ethernet packet. But of course if you want TCP/IP you will need to supply your own software implementation instead.
I was recently bringing up a custom board with a W6100 chip and it’s funny how closely my experience paralleled this one.
I also had (a different) issue with SPI communication (the W6100 uses a little different protocol).
I still couldn’t get any Ethernet packets to send/receive with a real machine, so I did basically the same debugging step: bought an off the shelf W6100 dev board and planned to use their arduino library to compare against. That’s where I left that project.
is there already a "networking from scratch" type of content somewhere -- not focussed on implementing it in hardware -- but maybe a tool for interactively playing with bits, bytes, packets, frames, etc to simulate each layer of the stack and see how 2 or more devices on a network including switches and routers might communicate with each other
I remember many many years ago, learning about networking, I got very fixated towards constructing and sending my own packet over the wire. Good times.
How quickly can you get from tinkering with Ethernet on Arduino to e.g. creating 10Gbit router with FPGA?
I don't want to sound bad, but it's a bit like writing BASIC programs on C64 today? Fun, entertaining, but is it actually useful for developing skills in this domain?
> Due to the complexity of the signalling involved with ethernet, a dedicated ASIC is generally used, which takes in data at the frame level, and takes care of wiggling the electrical lines in the right way over a cable (though there are of course exceptions).
Like others in this thread I was hoping I’d see an RJ45 breakout board directly hooked into an STM32 and doing onerous bit banging and keeping it as low level and basic as possible.
I sent an Ethernet packet
(github.com)416 points by todsacerdoti 11 November 2024 | 126 comments
Comments
> Strangely, I've found that there is a not-insignificant number of people are against this - especially in a professional environment. I think this has something to do with the JIRA-fication of development process, where all work is divided and sub-divided, and anything that does not directly check a box and produce a deliverable is wasted effort. It's disheartening that there is a whole group of people - developers and managers alike - who do not understand or see the value in exploration as part of the process.
So true, being able to create your own small tools is a superpower which is often at the heart of 10x programmers. It is sadly an art often practiced in the shadow.
(I was hoping it would be about bit-banging Ethernet on some improbable bit of hardware.)
Also,
> Due to the complexity of the signalling involved with ethernet, a dedicated ASIC is generally used
In the context of microcontrollers, I think this is generally not true. In most cases Ethernet functionality is incorporated as a embedded peripheral within the microcontroller itself, as it is with the STM32F407 and the ESP32 (both of which happen to use the same Ethernet peripheral sourced from an external provider).
Packets are an IP concept :)
Their 'STM32Cube' software gets mixed reviews, but it will spit out a working example of ethernet communication.
[1] https://www.st.com/en/evaluation-tools/nucleo-f439zi.html
It took me a while to realize the data consumers (those who need to look at and understand data) didnt have any orchestration tools, except a creaky UI cron type of scheduler.
For my work I refused to compromise and spend all my days clicking in a UI, so I searched around for a good tool, decided on Prefect. It served me well, but I didn't realize going in that Prefect/Airflow/Argo etc are really orchestration engines and you still need to write the client software around it to serve your own (or the team) productivity.
For example, connections to sources, methods to extract from source with a sql file, where to put the data after, injecting into publication channels, etc. I all gradually ended up writing functions to support this for my own personal IC work. I sank a ton of time into learning, had to learn how to develop python packages (I chose poetry), on an on. And yet despite all that time spent grinding away on indirectly productive learning I have still been more productive than my peers.
It was a mystery so I spent time learning about my peer workflows, which people are so cagey about at my company. Anyway, everybody just crams 100s of lines of business logic into Tableau custom sql sources to avoid scrutiny of data artifacts like tables they might create on the data lake. I guess these are tableau flavored views, but its so hard to read and understand all this logic in the UI -- oh and the calculated fields also.
I guess to sum up, if I am keeping up with peers using the latest expensive enterprise 'data storytelling' service while I self educate on basically data engineering to stats presentation plots, spark, julia, python, etc. I think I have concluded that Tableau should be considered harmful.
When it comes to tcp connections linux seems more robust.
[1] https://www.kernel.org/doc/html/v6.9/networking/af_xdp.html
I was going to implement TCP, but got bored (it suddenly becomes way more difficult)...
It's just so amazing to me to behold the entire network stack and the fact it actually works (even if IPv4 is clinging on at this stage).
The moment I received my first packet on a cut-up wired headset I used as a transceiver makeshift tenceiver it felt like something clicked and I just began to understand more how the universe works. I recommend projects of this type and wish more folks did it.
I wrote a little framework (bootloader, utilities, etc.) to allow for updating firmware and tunneling stdout on AVRs over pure Ethernet.
https://github.com/jakemoroni/Micro-Protocol-SDK
I also had (a different) issue with SPI communication (the W6100 uses a little different protocol).
I still couldn’t get any Ethernet packets to send/receive with a real machine, so I did basically the same debugging step: bought an off the shelf W6100 dev board and planned to use their arduino library to compare against. That’s where I left that project.
I admire this kind of low-level, first principles exploration which I personally don't do enough of.
I'll never forget Packetfu (https://rubygems.org/gems/packetfu)
I don't want to sound bad, but it's a bit like writing BASIC programs on C64 today? Fun, entertaining, but is it actually useful for developing skills in this domain?
It was really fun and surprisingly helped me get ahead in the networking class I'm taking.
Like others in this thread I was hoping I’d see an RJ45 breakout board directly hooked into an STM32 and doing onerous bit banging and keeping it as low level and basic as possible.
For Consultation, Contact Us Now! Website - https://www.bizex.ae/ Email - info@bizex.ae Call - +971 444 73414 Address - Office No-304, Al Mankhool Building (BMI Building), Khalid Bin Al Waleed, Bur Dubai, Dubai