As my colleague Scott says, TCP is required for RDMnet, but only at most one TCP connection is required at a time. I would recommend looking into the compile-time config options of your network stack and see if you can tweak the settings to just the minimum necessary to support a single TCP connection. For example, in the popular stack lwIP, there are many config options that control how much memory is allocated to hold state for connections; most of them are defined in
this file.
As you probably know, 64K flash and 8K ram is toward the bottom range of hardware that can support any kind of network stack at all. It may not be practical to support RDMnet on this hardware, but even if not, you almost certainly could do it with a spec increase on the same order of magnitude (say, 128K flash, 32K ram).