@Gertjan
In my last packet capture I used another device, therefore the mac address difference.
The Lease is denied upon entering bound looked weird to me, as the .53 is the PXE server and it should not be responding to any DHCP request since the gateway .1 has to. What we did next is made sure I block inbound port 67 on the .53 (PXE server).
Then packet capture looked liked this:
10:33:09.366450 a8:2b:dd:6b:80:5f > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 389: (tos 0x0, ttl 64, id 57999, offset 0, flags [none], proto UDP (17), length 375)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from a8:2b:dd:6b:80:5f, length 347, xid 0x8463ff2, Flags [Broadcast] (0x8000)
Client-Ethernet-Address a8:2b:dd:6b:80:5f
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
MSZ (57), length 2: 1472
Parameter-Request (55), length 35:
Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4)
IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13)
Domain-Name (15), RP (17), EP (18), RSZ (22)
TTL (23), BR (28), YD (40), YS (41)
NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51)
Server-ID (54), RN (58), RB (59), Vendor-Class (60)
TFTP (66), BF (67), GUID (97), Unknown (128)
Unknown (129), Unknown (130), Unknown (131), Unknown (132)
Unknown (133), Unknown (134), Unknown (135)
GUID (97), length 17: 0.0.158.211.19.126.180.240.17.128.191.159.198.84.14.38.0
NDI (94), length 3: 1.3.16
ARCH (93), length 2: 7
Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016"
10:33:09.369504 90:ec:77:94:34:37 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 357: (tos 0x10, ttl 128, id 0, offset 0, flags [DF], proto UDP (17), length 343)
xxx.xxx.xxx.1.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 315, xid 0x8463ff2, Flags [Broadcast] (0x8000)
Your-IP xxx.xxx.xxx.30
Server-IP xxx.xxx.xxx.53
Client-Ethernet-Address a8:2b:dd:6b:80:5f
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: xxx.xxx.xxx.1
Domain-Name-Server (6), length 4: xxx.xxx.xxx.7
Domain-Name (15), length 6: "mydomain"
Lease-Time (51), length 4: 7200
Server-ID (54), length 4: xxx.xxx.xxx.1
BF (67), length 31: "\boot\efiboot\amd64\bootx64.efi"
10:33:09.373175 00:0c:29:e0:55:2c > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 322: (tos 0x0, ttl 128, id 45651, offset 0, flags [none], proto UDP (17), length 308)
xxx.xxx.xxx.53.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 280, xid 0x8463ff2, Flags [Broadcast] (0x8000)
Server-IP xxx.xxx.xxx.53
Client-Ethernet-Address a8:2b:dd:6b:80:5f
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
GUID (97), length 17: 0.0.158.211.19.126.180.240.17.128.191.159.198.84.14.38.0
Vendor-Class (60), length 9: "PXEClient"
Server-ID (54), length 4: xxx.xxx.xxx.53
10:33:12.693431 a8:2b:dd:6b:80:5f > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 395: (tos 0x0, ttl 64, id 58000, offset 0, flags [none], proto UDP (17), length 381)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from a8:2b:dd:6b:80:5f, length 353, xid 0x8463ff2, Flags [Broadcast] (0x8000)
Client-Ethernet-Address a8:2b:dd:6b:80:5f
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Request
Server-ID (54), length 4: xxx.xxx.xxx.53
MSZ (57), length 2: 65280
Parameter-Request (55), length 35:
Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4)
IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13)
Domain-Name (15), RP (17), EP (18), RSZ (22)
TTL (23), BR (28), YD (40), YS (41)
NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51)
Server-ID (54), RN (58), RB (59), Vendor-Class (60)
TFTP (66), BF (67), GUID (97), Unknown (128)
Unknown (129), Unknown (130), Unknown (131), Unknown (132)
Unknown (133), Unknown (134), Unknown (135)
GUID (97), length 17: 0.0.158.211.19.126.180.240.17.128.191.159.198.84.14.38.0
NDI (94), length 3: 1.3.16
ARCH (93), length 2: 7
Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016"
10:33:12.698998 00:0c:29:e0:55:2c > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 406: (tos 0x0, ttl 128, id 45652, offset 0, flags [none], proto UDP (17), length 392)
xxx.xxx.xxx.53.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 364, xid 0x8463ff2, Flags [Broadcast] (0x8000)
Server-IP xxx.xxx.xxx.53
Client-Ethernet-Address a8:2b:dd:6b:80:5f
file "a8-2b-dd-6b-80-5f\boot\efiboot\amd64\bootx64.efi"
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: ACK
GUID (97), length 17: 0.0.158.211.19.126.180.240.17.128.191.159.198.84.14.38.0
Vendor-Class (60), length 9: "PXEClient"
Server-ID (54), length 4: xxx.xxx.xxx.53
Unknown (200), length 4: 1669485411
Unknown (201), length 46: 360,29812,28730,12079,25456,29541,29302,25970,11873,27757,25976,24878,27759,25441,27706,13616,13104,12131,26991,29540,25968,27759,30976
Unknown (202), length 28: 795240812,1702047585,1886415977,1667331177,1869492079,1935958892,1768255092
10:33:12.699037 a8:2b:dd:6b:80:5f > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 329: (tos 0x0, ttl 64, id 58001, offset 0, flags [none], proto UDP (17), length 315)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from a8:2b:dd:6b:80:5f, length 287, xid 0x8463ff2, Flags [Broadcast] (0x8000)
Client-Ethernet-Address a8:2b:dd:6b:80:5f
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Decline
Server-ID (54), length 4: xxx.xxx.xxx.53
MSG (56), length 35: "Lease is denied upon entering bound"
That much with this theory. Since it was already urgent and we had to make it work, what we did next was go to Firewall --> DHCP server --> (Click on our VLAN) --> and check the box next to Do not record a unique identifier (UID) in client lease data if present in the client DHCP request, then Status --> Services --> Restart kea-dhcp4 service
Now everything works as it should, here is a packet capture from a working PXE boot:
10:48:13.603483 a8:2b:dd:6b:80:5f > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 401: (tos 0x0, ttl 64, id 25450, offset 0, flags [none], proto UDP (17), length 387)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from a8:2b:dd:6b:80:5f, length 359, xid 0x27ce53d9, Flags [Broadcast] (0x8000)
Client-Ethernet-Address a8:2b:dd:6b:80:5f
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Request
Server-ID (54), length 4: xxx.xxx.xxx.1
Requested-IP (50), length 4: xxx.xxx.xxx.30
MSZ (57), length 2: 65280
Parameter-Request (55), length 35:
Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4)
IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13)
Domain-Name (15), RP (17), EP (18), RSZ (22)
TTL (23), BR (28), YD (40), YS (41)
NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51)
Server-ID (54), RN (58), RB (59), Vendor-Class (60)
TFTP (66), BF (67), GUID (97), Unknown (128)
Unknown (129), Unknown (130), Unknown (131), Unknown (132)
Unknown (133), Unknown (134), Unknown (135)
GUID (97), length 17: 0.0.158.211.19.126.180.240.17.128.191.159.198.84.14.38.0
NDI (94), length 3: 1.3.16
ARCH (93), length 2: 7
Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016"
10:48:13.610271 90:ec:77:94:34:37 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 357: (tos 0x10, ttl 128, id 0, offset 0, flags [DF], proto UDP (17), length 343)
xxx.xxx.xxx.1.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 315, xid 0x27ce53d9, Flags [Broadcast] (0x8000)
Your-IP xxx.xxx.xxx.30
Server-IP xxx.xxx.xxx.53
Client-Ethernet-Address a8:2b:dd:6b:80:5f
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: ACK
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: xxx.xxx.xxx.1
Domain-Name-Server (6), length 4: xxx.xxx.xxx.7
Domain-Name (15), length 6: "mydomain"
Lease-Time (51), length 4: 7200
Server-ID (54), length 4: xxx.xxx.xxx.1
BF (67), length 31: "\boot\efiboot\amd64\bootx64.efi"
10:48:13.620307 00:0c:29:e0:55:2c > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 406: (tos 0x0, ttl 128, id 45659, offset 0, flags [none], proto UDP (17), length 392)
xxx.xxx.xxx.53.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 364, xid 0x27ce53d9, Flags [Broadcast] (0x8000)
Server-IP xxx.xxx.xxx.53
Client-Ethernet-Address a8:2b:dd:6b:80:5f
file "a8-2b-dd-6b-80-5f\boot\efiboot\amd64\bootx64.efi"
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: ACK
GUID (97), length 17: 0.0.158.211.19.126.180.240.17.128.191.159.198.84.14.38.0
Vendor-Class (60), length 9: "PXEClient"
Server-ID (54), length 4: xxx.xxx.xxx.53
Unknown (200), length 4: 1669485411
Unknown (201), length 46: 360,29812,28730,12079,25456,29541,29302,25970,11873,27757,25976,24878,27759,25441,27706,13616,13104,12131,26991,29540,25968,27759,30976
Unknown (202), length 28: 795240812,1702047585,1886415977,1667331177,1869492079,1935958892,1768255092
Regarding the IP addresses, yes they are RFC1918 (but I have to sanitize the output either way, story for another time).
Any explanation as to why this may happen? Possibly a bug with KEA DHCP?