DNS Settings

[howardswe499]

OS: Ubuntu 22.04.4 LTS 5.15.0-97-generic

Hello again @linouxis9 ,

I am trying to let PR use tunnel as its default GW. So, after connecting PR to 5GC, I edited my routing table manually to change from this:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.102.254 0.0.0.0         UG    0      0        0 ens192
10.246.112.0    192.168.102.254 255.255.255.0   UG    0      0        0 ens192
172.16.13.0     0.0.0.0         255.255.255.0   U     0      0        0 ens224
192.168.96.0    192.168.102.254 255.255.252.0   UG    0      0        0 ens192
192.168.102.0   0.0.0.0         255.255.255.0   U     0      0        0 ens192

to this

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 vrf9000000007
10.246.112.0    192.168.102.254 255.255.255.0   UG    0      0        0 ens192
172.16.13.0     0.0.0.0         255.255.255.0   U     0      0        0 ens224
192.168.96.0    192.168.102.254 255.255.252.0   UG    0      0        0 ens192
192.168.102.0   0.0.0.0         255.255.255.0   U     0      0        0 ens192

Using these routing settings, I can ping 8.8.8.8 using ping 8.8.8.8 no problem. But ping www.google.com does not work, and I suspect is due to missing DNS server config. So far, both PR and UERANSIM get stuck on this step.

On UERANSIM, I am able to workaround:

Workaround 1:
Add nameserver 8.8.8.8 to /etc/resolv.conf
This only fixes temporarily.

# ping www.google.com
PING www.google.com (172.217.163.36) 56(84) bytes of data.
64 bytes from tsa01s13-in-f4.1e100.net (172.217.163.36): icmp_seq=1 ttl=115 time=17.6 ms
64 bytes from tsa01s13-in-f4.1e100.net (172.217.163.36): icmp_seq=2 ttl=115 time=17.4 ms
...

Workaround 2:
Add the following to /etc/netplan/*.yaml (i.e. editing network settings)

    uesimtun0: # UERANSIM tunnel
      dhcp4: true
      nameservers:
        addresses:
        - 8.8.8.8

Apply config

# sudo netplan apply

=== After setting up default GW ===

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=17.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=18.5 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 17.888/18.175/18.462/0.287 ms
# ping www.google.com
PING www.google.com (142.251.43.4) 56(84) bytes of data.
64 bytes from tsa03s08-in-f4.1e100.net (142.251.43.4): icmp_seq=1 ttl=56 time=17.7 ms
64 bytes from tsa03s08-in-f4.1e100.net (142.251.43.4): icmp_seq=2 ttl=56 time=17.9 ms
64 bytes from tsa03s08-in-f4.1e100.net (142.251.43.4): icmp_seq=3 ttl=56 time=17.1 ms
^C
--- www.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 17.120/17.573/17.899/0.330 ms
# nslookup www.google.com
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   www.google.com
Address: 142.251.43.4
Name:   www.google.com
Address: 2404:6800:4012:2::2004

On PR, however, only Workaround 1 works, and it only works if specified to use vrf interface. ping -I vrf9000000007 www.google.com

Workaround 2 does not work for PR, using the following settings in /etc/netplan/*.yaml:

    vrf9000000007: # via PR
      dhcp4: true
      nameservers:
        addresses:
        - 8.8.8.8

Apply config

# sudo netplan apply

=== After setting up default GW ===

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=16.4 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=16.8 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=17.3 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=16.4 ms
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 16.383/16.728/17.314/0.383 ms
# ping www.google.com
ping: www.google.com: Temporary failure in name resolution
# nslookup www.google.com 
Server:         127.0.0.53
Address:        127.0.0.53#53

** server can't find www.google.com: SERVFAIL

I have two questions:

1. I'm not sure why DNS information from 5GC is not accepted by UE simulators. I checked the PDU Session Establishment Accept packet and I see DNS info being received.
2. Why Workaround 2 does not work for PR.

I would be very grateful if you could shed some light on this situation.

Thanks in advance!
Howard

[linouxis9]

Hi @howardswe499,

I hope you are doing well!
#69 should have all the information you need on this issue.
If you use only a single UE, you may add --tunnel-vrf false to workaround the issue with WA#2 as well.
Also by the way, it is expected in PR that you have to mention the VRF for the traffic to go through PacketRusher, it is to avoid issue with your other computer's traffic and allowing you to have multiple UEs running at the same on your computer. The route is automatically created inside the VRF.
You can tweak this behavior with --tunnel-vrf as well, or ip vrf exec vrfXXXXX bash and then you have a shell where all commands will use the VRF.
Feel free to get back to me if needed!

Thanks and cheers,
Valentin

[howardswe499]

Hi @linouxis9 ,

Thanks for the information.

I tried running a combination of commands with --tunnel-vrf false, but they all seemed to either ignore the false argument or have problems with recognizing the options.

Here is what I tried:

# ./packetrusher multi-ue -n 1 --tunnel --dedicatedGnb --tunnel-vrf false
# ./packetrusher multi-ue -n 1 --dedicatedGnb --tunnel --tunnel-vrf false

The above commands outputted the following (ignored argument)

===============
=== PR Logs ===
===============
INFO[0000] Selecting 10.246.112.18 for host 10.246.112.18 as AMF's IP address 
INFO[0000] Selecting 192.168.102.54 for host 192.168.102.54 as gNodeB's N3/Data IP address 
[trimmed]
INFO[0002] [UE][NAS] PDU address received: 10.246.115.10 
INFO[0003] [UE][GTP] Interface val9000000007 has successfully been configured for UE 10.246.115.10 
INFO[0003] [UE][GTP] You can do traffic for this UE using VRF vrf9000000007, eg: 
INFO[0003] [UE][GTP] sudo ip vrf exec vrf9000000007 iperf3 -c IPERF_SERVER -p PORT -t 9000 

===================
=== ip vrf show ===
===================
root@da-ueransim:~# ip vrf show
Name              Table
-----------------------
vrf9000000007    1075839253

# ./packetrusher multi-ue -n 1 --tunnel-vrf false
# ./packetrusher multi-ue -n 1 --tunnel-vrf false --dedicatedGnb --tunnel
# ./packetrusher multi-ue -n 1 --tunnel-vrf false --tunnel --dedicatedGnb

The above commands outputted the following (cannot recognize options)

===============
=== PR Logs ===
===============
INFO[0000] Selecting 10.246.112.18 for host 10.246.112.18 as AMF's IP address 
INFO[0000] Selecting 192.168.102.54 for host 192.168.102.54 as gNodeB's N3/Data IP address 
[trimmed]
INFO[0002] [UE][NAS] PDU address received: 10.246.115.5 
INFO[0002] [UE][GTP] Interface for UE 9000000007 has not been created. Tunnel has been disabled. 

=============================
=== Tunnel interface test ===
=============================
root@da-ueransim:~# ping -I val9000000007 8.8.8.8
ping: SO_BINDTODEVICE val9000000007: No such device

Additionally, after going through the thread you provided, I'm afraid I still don't understand why 5GC DNS server info is not imported (?) as DNS for the vrf/tunnel interface. (i.e. is there a way to allow for automatic importation of DNS info)

Thanks
Howard

[howardswe499]

Following the article linked in the thread & the tips you provided, I managed to devise an improvement to WA#1 that fixes the persistency:

1. Stop & disable systemd-resolved.service

# systemctl stop systemd-resolved.service
# systemctl disable systemd-resolved.service

2. Replace /etc/resolv.conf with nameservers of /run/systemd/resolve/resolv.conf

# grep nameserver /run/systemd/resolve/resolv.conf > /etc/resolv.conf

3. Start PR normally
4. Create bash session that uses vrf vrf9000000007

# ip vrf exec vrf9000000007 bash

This enables me to use custom nameservers without fear of resolv.conf being overwritten by systemd-resolved, which is good enough for now!

[linouxis9]

Hi @howardswe499!

I tried running a combination of commands with --tunnel-vrf false, but they all seemed to either ignore the false argument or have problems with recognizing the options.

We indeed found a bug in the parsing of the parameters, we'll fix it, thanks for the report as usual! :-)

Additionally, after going through the thread you provided, I'm afraid I still don't understand why 5GC DNS server info is not imported (?) as DNS for the vrf/tunnel interface. (i.e. is there a way to allow for automatic importation of DNS info)

PacketRusher first goal is to be able to be ran on a developer's laptop or in a VM, and as such, modifying system-wide DNS and make ALL the traffic of the system goes through the tunnel, seems out of place. While it would be fine in a container (not completely, since you would lose access to k8s's CoreDNS and the ability to resolve other pods' FQDN), you would have the same issue if you had two UEs receiving distinct DNS servers in your PacketRusher's container.
The issue mostly lies in the fact that you cannot set a DNS server per VRF, if we could do that, we could safely modifythe DNS server to be used by the UEs without impacting others processes of PacketRusher's host.
One of the alternative would be to use network namespaces instead of VRF, but there are other limitations regarding network namespaces, and you cannot create an unlimited number of them without tweaking ucount.

Hope it helped!

Thanks and cheers,
Valentin

[howardswe499]

Hi @linouxis9 ,

Thank you for your explanation, DNS does seem quite cumbersome to implement. I was coming from the perspective of using PR as a way to allow any device running Linux to connect to a private 5GC that is part of the LAN of say, a corporation. The ability to accept custom DNS servers provided by the private 5GC could be useful, I think, such as in scenarios where applications running in the LAN use FQDN rather than IP for connectivity.

That does bring into question, however, as to why not just use the existing wi-fi or wired infrastructure instead 😆

Oh well, it was an interesting discussion. Thanks for providing insight into this problem that has been bugging me for a while, even if ultimately it wasn't very fruitful.

Feel free to let me know if you need further testing.

Cheers
Howard

[linouxis9]

Hi @howardswe499,

Ah yes, I understand your use case! Though I think for this use case, it would be better to use something like N3IWF + N3IWUE or TNGF which are basically the 3GPP way of accessing a private 5GC connected to a LAN.
However, a N3IWF works very similarly as a gNodeB, and it could potentially be interesting to add support for N3IWF UEs inside PacketRusher!

That does bring into question, however, as to why not just use the existing wi-fi or wired infrastructure instead 😆

This is a very good point, and in my experience, companies selling P5GC usually have difficulties answering this question :D

Oh well, it was an interesting discussion. Thanks for providing insight into this problem that has been bugging me for a while, even if ultimately it wasn't very fruitful.

Happy to help! Feel free to open a discussion whenever you want if you want to discuss about something!

Feel free to let me know if you need further testing.

Will do!

Thanks!
Valentin

[linouxis9]

Hi @howardswe499,

After thinking a bit, I think I might have an idea. It's not totally clean but could be doable once we switch to the eBPF GTP-U data path #47 to replace the gtp5g module.
For instance, we could try to NAT all DNS traffic sent to 127.0.0.53 to the DNS server sent by the 5GC using the eBPF program already attached inside the VRF.
We already inspect packets inside it for encapsulation/decapsulation, so speed should not be impacted too much that way.

Cheers,
Valentin

[howardswe499]

Sounds exciting, hope to see the eBPF version release soon!

BTW, do you know if packet capture will still be possible via tcpdump after the change? On eUPF, I noticed that interfaces used by eBPF seems to intercept the packets before it reaches the Linux kernel, leading to tcpdump being unable to capture the packets on the interface. I had to resort to capturing from the other side of the N3/N4/N6 interfaces to inspect the packets being sent, and inspecting XDP logs for clues as to how eUPF was forwarding the packets.

Howard

[linouxis9]

Sounds exciting, hope to see the eBPF version release soon!

Sadly, I don't have much time to work on PacketRusher lately, so I've been working mostly on issues and smaller tasks, I wouldn't count on the eBPF version be released soon :( If any contributors reading this wanna hop on the wagon and contribute this, please do so :D

BTW, do you know if packet capture will still be possible via tcpdump after the change? On eUPF, I noticed that interfaces used by eBPF seems to intercept the packets before it reaches the Linux kernel, leading to tcpdump being unable to capture the packets on the interface. I had to resort to capturing from the other side of the N3/N4/N6 interfaces to inspect the packets being sent, and inspecting XDP logs for clues as to how eUPF was forwarding the packets.

Yes, this is indeed an eBPF limitation that'll also apply to PacketRusher. You may use xdpdump instead of tcpdump to capture packets before being intercepted by the eBPF program.
On PacketRusher, with tcpdump, you'll only be able to observe downlink decapsulated traffic on UE's interface and uplink GTP-U encapsulated traffic on N3 interface IIRC.
Also, interesting to mention, PacketRusher's eBPF version is pretty much just eUPF's eBPF program being integrated inside PacketRusher and slightly tweaked to work for gNodeB's usecases. You can try an alpha here but it's based upon an old version of PacketRusher that may have some bugs.

Valentin