Trying to connect to POE cameras remotely through 4G router

[GuillaumeHupi]

Hello,

We are using OAK-1 PoE cameras. They are deployed on site and connected via a 4G router. This all works, but if we need to do a software update, for now we go on site, connect to the router to be on the same network as the camera and run the python script to flash a firmware in standalone mode.

We will have to deploy the solution on sites quite far away and we are looking for a way to be able to make updates remotely. Our telephone operator provides us with VPN access and the IP of our SIM card once connected to them in VPN in the format 10.X.X.X. Thus, we can for example connect to the router remotely using this IP.

On our router we can also configure a port redirection, So for example, if we connect a computer to the router and if we configure a port redirection for port 22 on the computer's IP. Then, we can remotely access this computer in ssh using the SIM IP. For example if the IP of my SIM is 10.10.10.25. If I connect to the operator's VPN and I run a command "ssh root@10.10.10.25", I am well connected in ssh to my computer connected by ethernet to the 4G router.

So we were able to validate that the port redirection works well. We then wanted to see if it was possible to access the POE camera with the same method.

In doubt, on the router we configured port forwarding for all ports (0-65535) to the camera IP. Because we analyzed the requests made and several ports are used (at least 90 for ping and 11490 for firmware flash).

And then, we just tried to launch the command

dai.DeviceBootloader.getFirstAvailableDevice()

But it could not find a device. We thought it was related to the fact that the camera IP is not on the same network mask as the computer on which we are running the program (camera in 10.X.X.X, while the computer is in 192.168.1.X). So we wanted to specify the IP through this command:

bootloader = depthai.DeviceBootloader(ip_address, False)

But remotely, it is impossible to detect the camera:

bootloader = depthai.DeviceBootloader(ip_address, False)
RuntimeError: Specified device not found

But we are able to ping the camera on the 10.X.X.X IP so I don't uderstand why the python script is not able to find it.

Do you have any idea what causes this behavior? For example, does the python library encounter a problem when we try to use an IP that is on a different address range than the computer on which the script is executed? Or is there a particular setting that we should use on our router?

[themarpe]

@GuillaumeHupi (CC: @jakaskerl )

You'll need to check if UDP 11491 is also passed through as that is required for both "device discovery" as well as when specific IP is set (for device state check)

[GuillaumeHupi]

@themarpe ,

Sorry for my late response.
We made different tests. I don't know why; but when forwarding all ports, it wasn't working, but when forwarding only 11491 and 11490 ports on our router, we are able to detect the device remotely through the VPN.

But we are not able to flash a new script remotely. I found this bug luxonis/depthai-core#794 and we are in exaclty the same scenario and with the same result.

I am not sure I understood what was said in the discussion. From what I understood, the problem is caused by the bootloader that is not reseted when we flash firmware to the camera. And so it is triggered before the flash is done and the camera reboot which stops the flash process before the end and causes an error. And this is due to 4G latency which make the process longer than via direct ethernet. Am I wrong ?

If I'm right, a trick to make it work would be to extend the watchdog or to totally disable it. I tried to flash my script while specifying the the watchdog and bootup delays like this :
DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 DEPTHAI_BOOTUP_TIMEOUT=60000 python oak_detection_lifeline.py

Or to totally disable the watchdog like this :
DEPTHAI_WATCHDOG=0 python oak_detection_lifeline.py

But I still got the same problem. In the github discussion, it was mentioned to also make some modifications to do inside the C library, but I don't know where to find this library when I use the depthai-python library inside a script.

So I'm a bit lost. Could you explain to me the detailed procedure on how to disable the watchdog, in order to make tests and be sure that this is the watchdog that causes us this problem ?

Thanks

Guillaume

[themarpe]

CC: @moratom if you could help @GuillaumeHupi with a artifacts of a built python package against latest + added this patch on top (from the other issue's thread): luxonis/depthai-core@main...mrmorawski:depthai-core:no_bootloader_watchdog and or we bring it in as DEPTHAI_BOOTLOADER_WATCHDOG env var & bake it in. (note, naive impl. won't work fully, as it doesn't change device's WD, but BL has an internal bump per any "comms received" so its should work for most cases)

[moratom]

@GuillaumeHupi I've added internal tracking on this and will update the case once we have the artifact build ready.