USB data issue

[stefan981]

Hey Marek,

Thanks for the great app; it has helped me a lot with development.

Two months ago, the USB connection from my iPad worked, and I was able to stream the RGB and the depth image. However, now when I try to do it with the same code, it does not work. I get an OpenCV assertion that states the RGB matrix is empty. When I only use the depth image, I don't receive any errors, but the output is just a black image.

I have installed the latest app version and pulled the latest code from GitHub. The behavior is the same for both the Python and C++ versions. It is able to start the USB connection, the software recognises the device and starts the streaming.

Is this a known issue? I couldn’t find any relevant issues about this. Do you have any suggestions on what my problem could be?

Update:
It seems that the resize funbction in get_rgb_frame() causes the issues. This is the error:
ValueError: resize only works on single-segment arrays
Force restart the IPad like suggested in #71 did not help.

Best wishes,
Stefan

[marek-simonik]

Hi Stefan,

thank you for using Record3D.

I could not replicate the problem with the following setup: Record3D 1.10.2 (the latest version), iOS 17.6.1, Ubuntu, Python 3.8.

May I ask what is your setup?

Looking forward to fixing this!
Marek

[marek-simonik]

I was able to replicate the problem with Python 3.10.

Please clone record3d, replace this highlighted section of Record3DStream.h by the snippet below, build/install the record3d library and let me know if you still get an error.

/**
 * NOTE: This is alternative API for Python.
 *
 * @returns the current contents of the Depth buffer which holds the lastly received Depth frame.
 */
py::array_t<float> GetCurrentDepthFrame()
{
    size_t currentFrameWidth = currentFrameDepthWidth_;
    size_t currentFrameHeight = currentFrameDepthHeight_;

    size_t bufferSize  = currentFrameWidth * currentFrameHeight * sizeof(float);
    auto result        = py::array_t<float>({ static_cast<int>(currentFrameHeight), static_cast<int>(currentFrameWidth) });
    auto result_buffer = result.request();
    float *result_ptr  = (float *) result_buffer.ptr;

    std::memcpy(result_ptr, depthImageBuffer_.data(), bufferSize);

    return result;
}

/**
 * NOTE: This is alternative API for Python.
 *
 * @returns the current contents of the Confidence buffer which holds the lastly received Depth frame.
 */
py::array_t<uint8_t> GetCurrentConfidenceFrame()
{
    size_t currentFrameWidth = currentFrameConfidenceWidth_;
    size_t currentFrameHeight = currentFrameConfidenceHeight_;

    size_t bufferSize  = currentFrameWidth * currentFrameHeight * sizeof(uint8_t);
    auto result        = py::array_t<uint8_t>({static_cast<int>(currentFrameHeight), static_cast<int>(currentFrameWidth)});
    auto result_buffer = result.request();
    uint8_t *result_ptr  = (uint8_t *) result_buffer.ptr;

    std::memcpy(result_ptr, confidenceImageBuffer_.data(), bufferSize);

    return result;
}

/**
 * NOTE: This is alternative API for Python.
 *
 * @returns the current contents of the Misc buffer (reserved).
 */
py::array_t<uint8_t> GetCurrentMiscData()
{
    size_t bufferSize  = miscBuffer_.size();
    auto result        = py::array_t<uint8_t>( bufferSize );
    auto result_buffer = result.request();
    uint8_t *result_ptr  = (uint8_t *) result_buffer.ptr;

    std::memcpy(result_ptr, miscBuffer_.data(), bufferSize);

    return result;
}

/**
 * NOTE: This is alternative API for Python.
 *
 * @returns the current contents of the RGB buffer which holds the lastly received RGB frame.
 */
py::array_t<uint8_t> GetCurrentRGBFrame()
{
    size_t currentFrameWidth = currentFrameRGBWidth_;
    size_t currentFrameHeight = currentFrameRGBHeight_;

    constexpr int numChannels = 3;
    size_t bufferSize  = currentFrameWidth * currentFrameHeight * numChannels * sizeof(uint8_t);
    auto result        = py::array_t<uint8_t>({static_cast<int>(currentFrameHeight), static_cast<int>(currentFrameWidth), numChannels});
    auto result_buffer = result.request();
    uint8_t *result_ptr  = (uint8_t *) result_buffer.ptr;

    std::memcpy(result_ptr, RGBImageBuffer_.data(), bufferSize);

    return result;
}

[sleeplessTLV]

Hi,
I am having a similar issue (but not certain it is the SAME issue) with my installation - Windows 11, Python 3.12
The iPhone is connected via USB to the PC (I can see it in the windows explorer), yet the demo-main.py
returns "0 devices found".
Is it the same issue? if so, can the new version of record3d binary be built with a fix? I am having issues compiling it locally (Cmake does not recognize the VS Code 2022 version 17 I have)
Thanks!

[marek-simonik]

Hi @sleeplessTLV,

the "0 devices found" is most likely a different issue than what @stefan981 described. The problem you are facing happens because your iPhone does not allow the your computer to access it.

You need to install iTunes to get all necessary drivers installed (once installed, open the detail of your iPhone in iTunes to check that the connection works). Then, once you (re-)connect your iPhone to your computer via USB cable and unlock it, you will see the "Trust this Computer" pop-up. You will have to press "Trust" if you want to be able to stream via USB; this step will allow the communication between your iPhone and the computer (the library) to happen.

I am going to release a fixed version of the library with precompiled installers later this week. In the meantime, I can help you troubleshoot the compilation issues on your PC if you want — just let me know which exact command did you use to install the Python version of the record3d library from source. Try running either python setup.py install --user or, if that would not work, open Command Line or PowerShell as Administrator and try to install the library by python setup.py install. Feel free to send the error log either here or directly to my email: support@record3d.app

[sleeplessTLV]

Hi
Indeed upon installing itunes etc. I did get a device found, and the same error message as stefan981 (I use Python 3.12).
Running the python setup.py install from an Administrator command line I get
D:\ProgramData\Lidar\record3d>python setup.py install
running install
C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\setuptools_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

    ********************************************************************************
    Please avoid running ``setup.py`` directly.
    Instead, use pypa/build, pypa/installer or other
    standards-based tools.

    See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
    ********************************************************************************

!!
self.initialize_options()
C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\setuptools_distutils\cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

    ********************************************************************************
    Please avoid running ``setup.py`` and ``easy_install``.
    Instead, use pypa/build, pypa/installer or other
    standards-based tools.

    See https://github.com/pypa/setuptools/issues/917 for details.
    ********************************************************************************

!!
self.initialize_options()
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

[Errno 13] Permission denied: 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.12_3.12.2032.0_x64__qbz5n2kfra8p0\\Lib\\site-packages\\test-easy-install-2416.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.2032.0_x64__qbz5n2kfra8p0\Lib\site-packages\

Perhaps your account does not have write access to this directory? If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account. If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

Please make the appropriate changes for your system and try again.

[sleeplessTLV]

I tried also installing python 3.8, same error about permissions, and while it avoids the 'resize' error it has an error that the python 3.12 version did not report:
D:\ProgramData\Lidar\record3d>python3.8 demo-main.py
Traceback (most recent call last):
File "demo-main.py", line 2, in
from record3d import Record3DStream
ModuleNotFoundError: No module named 'record3d'

I assume this is since it simply got 'further' in the code before the error

[marek-simonik]

Python 3.12

Running the python setup.py install from an Administrator command line I get ...

Instead of running python setup.py install, you can try to install the record3d via pip like so:

python -m pip install .

Don't forget to include the trailing dot in the command!

If you don't have pip installed for one of your Python versions, then you should install it by downloading the get-pip.py script and then executing the get-pip.py script to install pip for the specific Python version you are executing the get-pip.py file with. If you want to install pip for both Python 3.8 and Python 3.12, then you will have to execute get-pip.py twice; i.e. once with the Python 3.8 executable and once with the Python 3.12 executable.

I am not sure why CMake does not see Visual Studio Build Tools on your system; please make sure that you do have the Visual Studio C++ Toolkit option installed on your computer and then restart.

Precompiled version of record3d with the fix above

If CMake would still not be able to find the C and C++ compilers on your computer, then as an alternative, I compiled the fixed version of record3d for Python 3.12 for you; I am attaching the .whl file, which you can install by running

python -m pip install record3d-1.4-cp312-cp312-win_amd64.whl

The precompiled package: record3d-prebuilt.zip

Python 3.8

I tried also installing python 3.8, same error about permissions, and while it avoids the 'resize' error it has an error that the python 3.12 version did not report:
D:\ProgramData\Lidar\record3d>python3.8 demo-main.py
Traceback (most recent call last):
File "demo-main.py", line 2, in
from record3d import Record3DStream
ModuleNotFoundError: No module named 'record3d'

I assume this is since it simply got 'further' in the code before the error

This error with Python 3.8 means that you did not install the record3d package for Python 3.8. To install record3d for Python 3.8, you should run python3.8 -m pip install record3d. I'm not sure if python3.8 is the correct executable for Python 3.8 on your computer, so make sure you know which Python version are you running.

Other

Also, don't forget to install OpenCV for Python to make the demo-main.py file run without errors:

python -m pip install opencv-python

Note that if you did not see any windows with live stream of images when running the C++ demo, then it means the C++ demo was not compiled with OpenCV support.

[sleeplessTLV]

Thanks for all the effort. Truly appreciated. I downloaded the whl file and ran the install command.
However, when I run python3.12 demo-main.py I still the same error as before of
ValueError: resize only works on single-segment arrays
I tried adding --force-reinstall, again no errors during the pip process but afterwords
still same error.
I suspect I need to replace manually some library file, but I don't know which

[marek-simonik]

Apologies, it looks like I mistakenly uploaded the old version of the library without the fix. Here is a link to precompiled record3d version 1.4.1, which should hopefully work now: record3d-1_4_1-prebuilt.zip

Please let me know whether reinstalling helped or not. Thanks.

[sleeplessTLV]

Hi,
yes it does work! thanks. I must say the cpp version (which I modified to save to disk as well) is 'smoother' somehow - frames are very consistent, while the python has some glitches in some frames, I think.
But it is great. thanks again!

[marek-simonik]

Thanks for confirming the latest .whl did fix the error.

My guess is that the (occasional?) non-smoothness of the stream might have something to do with the way multithreading is currently implemented in the library (i.e. how the raw data buffers received from the iDevice via USB are passed from the background thread to the main thread, which uses OpenCV for visualization). But since multithreading bugs (or "slightly wrong" behaviour) are difficult for me to reproduce and test, I am not sure if I will manage to reliably fix them.