make update-data is not working on Debian testing (Bullseye)

[dafula]

Hello,
I have troubles compiling dev branch on my Debian Bullseye machine. On Debian 10 Direwolf compiles fine. Here are the steps and the output:

cd /tmp
git clone https://github.com/wb2osz/direwolf.git
cd direwolf/
git checkout dev
mkdir build

cmake ..
-- The C compiler identification is GNU 10.2.0
-- The CXX compiler identification is GNU 10.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.28.0") 
-- Dire Wolf Version: 1.6.0-adae04b
-- Build type set to: Release
CMake system: Linux
-- Target architecture: x86_64
-- Use SSE SIMD instructions
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Checking for module 'libgps'
--   Found libgps, version 3.20
-- Found GPSD: /usr/lib/x86_64-linux-gnu/libgps.so
-- Checking for module 'hamlib'
--   Found hamlib, version 3.3
CMake Warning (dev) at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:273 (message):
  The package name passed to `find_package_handle_standard_args` (HAMLIB)
  does not match the name of the calling package (hamlib).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  cmake/modules/Findhamlib.cmake:55 (find_package_handle_standard_args)
  CMakeLists.txt:246 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found HAMLIB: /usr/lib/x86_64-linux-gnu/libhamlib.so  
-- Found ALSA: /usr/lib/x86_64-linux-gnu/libasound.so (found version "1.2.3.2") 
-- Checking for module 'libudev'
--   Found libudev, version 246
CMake Warning (dev) at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:273 (message):
  The package name passed to `find_package_handle_standard_args` (UDEV) does
  not match the name of the calling package (udev).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  cmake/modules/Findudev.cmake:68 (find_package_handle_standard_args)
  CMakeLists.txt:260 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found UDEV: /usr/lib/x86_64-linux-gnu/libudev.so  
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/direwolf/build

Then when executing make update-data:

Scanning dependencies of target data_rename
[  0%] Built target data_rename
Scanning dependencies of target download_symbolsx
[  7%] Creating directories for 'download_symbolsx'
[  7%] Performing download step (download, verify and extract) for 'download_symbolsx'
-- Downloading...
   dst='/tmp/direwolf/build/data/archive.tar'
   timeout='none'
-- Using src='http://www.aprs.org/symbols/symbolsX.txt'
-- [download 7% complete]
-- [download 13% complete]
-- [download 20% complete]
-- [download 27% complete]
-- [download 34% complete]
-- [download 41% complete]
-- [download 48% complete]
-- [download 54% complete]
-- [download 55% complete]
-- [download 61% complete]
-- [download 68% complete]
-- [download 75% complete]
-- [download 81% complete]
-- [download 88% complete]
-- [download 94% complete]
-- [download 100% complete]
-- Downloading... done
-- extracting...
     src='/tmp/direwolf/build/data/archive.tar'
     dst='/tmp/direwolf/build/data/download_symbolsx-prefix/src/download_symbolsx'
-- extracting... [tar xf]
CMake Error: Problem with archive_read_open_file(): Unrecognized archive format
CMake Error: Problem extracting tar: /tmp/direwolf/build/data/archive.tar
-- extracting... [error clean up]
CMake Error at download_symbolsx-stamp/extract-download_symbolsx.cmake:33 (message):
  error: extract of '/tmp/direwolf/build/data/archive.tar' failed


make[3]: *** [data/CMakeFiles/download_symbolsx.dir/build.make:113: data/download_symbolsx-prefix/src/download_symbolsx-stamp/download_symbolsx-download] Error 1
make[2]: *** [CMakeFiles/Makefile2:408: data/CMakeFiles/download_symbolsx.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:388: data/CMakeFiles/update-data.dir/rule] Error 2
make: *** [Makefile:218: update-data] Error 2

Something in the new versions of the building environment is breaking the updating of symbols, tocalls etc.
When I only compile Direwolf on Debian Bullseye everything seems fine - see the attached text file.

dw_compile_output.txt

Regards,
Dafcho Tabakov

[barjac]

But surely a package build should not try to access the web for information!
It is blocked on our build systems, all the sources need to be provided in the src.rpm or not at all.
How can this be turned off, as I am having a 'similar' issue in that the urls can't be accessed and the build fails.
I would like to download the relevant files and provide them in our own sources manually. Is this supported without hacking the CMakeLists.txt files?
Barry

[AG7GN]

I confirm the same behavior on direwolf master branch on RaspiOS Bullseye 64-bit.

Under Bullseye, make update-data downloads archive.tar. Here's an excerpt:

pi@buildpi:/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build $ make update-data
Scanning dependencies of target data_rename
[  0%] Built target data_rename
Scanning dependencies of target download_symbolsx
[  7%] Creating directories for 'download_symbolsx'
[  7%] Performing download step (download, verify and extract) for 'download_symbolsx'
-- Downloading...
   dst='/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar'
   timeout='none'
-- Using src='http://www.aprs.org/symbols/symbolsX.txt'
-- [download 5% complete]
-- [download 12% complete]
-- [download 18% complete]
-- [download 25% complete]
-- [download 31% complete]
-- [download 38% complete]
-- [download 57% complete]
-- [download 70% complete]
-- [download 83% complete]
-- [download 100% complete]
-- Downloading... done
-- extracting...
     src='/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar'
     dst='/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/download_symbolsx-prefix/src/download_symbolsx'
-- extracting... [tar xf]
CMake Error: Problem with archive_read_open_file(): Unrecognized archive format
CMake Error: Problem extracting tar: /media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar
-- extracting... [error clean up]
CMake Error at download_symbolsx-stamp/extract-download_symbolsx.cmake:33 (message):
  error: extract of
  '/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar'
  failed


make[3]: *** [data/CMakeFiles/download_symbolsx.dir/build.make:113: data/download_symbolsx-prefix/src/download_symbolsx-stamp/download_symbolsx-download] Error 1
make[2]: *** [CMakeFiles/Makefile2:408: data/CMakeFiles/download_symbolsx.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:388: data/CMakeFiles/update-data.dir/rule] Error 2
make: *** [Makefile:218: update-data] Error 2

The archive.tar file that is downloaded is a plain text file, so tar xf fails. I don't know why it has a .tar extension.

On RaspiOS Buster, the same command does this:

pi@nexus-ag7gn-2:~/tmp/direwolf/build $ make update-data
Scanning dependencies of target data_rename
[  0%] Built target data_rename
Scanning dependencies of target download_symbolsx
[  7%] Creating directories for 'download_symbolsx'
[  7%] Performing download step (download and verify) for 'download_symbolsx'
-- Downloading...
   dst='/home/pi/tmp/direwolf/build/data/symbolsX.txt'
   timeout='none'
-- Using src='http://www.aprs.org/symbols/symbolsX.txt'
-- [download 5% complete]
-- [download 11% complete]
-- [download 17% complete]
-- [download 23% complete]
-- [download 29% complete]
-- [download 35% complete]
-- [download 43% complete]
-- [download 55% complete]
-- [download 61% complete]
-- [download 67% complete]
-- [download 73% complete]
-- [download 92% complete]
-- [download 100% complete]
-- Downloading... done
[  7%] No patch step for 'download_symbolsx'
[ 15%] No update step for 'download_symbolsx'
[ 23%] No configure step for 'download_symbolsx'
[ 23%] No build step for 'download_symbolsx'
[ 23%] No install step for 'download_symbolsx'
[ 30%] No test step for 'download_symbolsx'
[ 30%] Completed 'download_symbolsx'
[ 30%] Built target download_symbolsx
Scanning dependencies of target download_tocalls
[ 30%] Creating directories for 'download_tocalls'
[ 38%] Performing download step (download and verify) for 'download_tocalls'
-- Downloading...
   dst='/home/pi/tmp/direwolf/build/data/tocalls.txt'
   timeout='none'
-- Using src='http://www.aprs.org/aprs11/tocalls.txt'
-- [download 8% complete]
-- [download 17% complete]
-- [download 27% complete]
-- [download 37% complete]
-- [download 47% complete]
-- [download 57% complete]
-- [download 87% complete]
-- [download 100% complete]
-- Downloading... done
[ 46%] No patch step for 'download_tocalls'
[ 46%] No update step for 'download_tocalls'
[ 46%] No configure step for 'download_tocalls'
[ 53%] No build step for 'download_tocalls'
[ 61%] No install step for 'download_tocalls'
[ 61%] No test step for 'download_tocalls'
[ 61%] Completed 'download_tocalls'
[ 61%] Built target download_tocalls
Scanning dependencies of target download_symbols-new
[ 69%] Creating directories for 'download_symbols-new'
[ 69%] Performing download step (download and verify) for 'download_symbols-new'
-- Downloading...
   dst='/home/pi/tmp/direwolf/build/data/symbols-new.txt'
   timeout='none'
-- Using src='http://www.aprs.org/symbols/symbols-new.txt'
-- [download 5% complete]
-- [download 12% complete]
-- [download 18% complete]
-- [download 25% complete]
-- [download 41% complete]
-- [download 61% complete]
-- [download 67% complete]
-- [download 87% complete]
-- [download 100% complete]
-- Downloading... done
[ 69%] No patch step for 'download_symbols-new'
[ 76%] No update step for 'download_symbols-new'
[ 84%] No configure step for 'download_symbols-new'
[ 84%] No build step for 'download_symbols-new'
[ 84%] No install step for 'download_symbols-new'
[ 92%] No test step for 'download_symbols-new'
[100%] Completed 'download_symbols-new'
[100%] Built target download_symbols-new
Scanning dependencies of target update-data
[100%] Built target update-data

No attempt to download archive.tar.

Buster uses cmake version 3.16.3. Bullseye uses version 3.18.4.

After running cmake .. in the build subdirectory (before running make update-data), I recursively search for grep -R archive.tar * and find nothing. Doing the same under Bullseye, there are lots of matches:

pi@buildpi:/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf $ mkdir -p build && cd build
pi@buildpi:/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build $ cmake ..
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.30.2") 
-- Dire Wolf Version: 1.6.0-fe6cba2
-- Build type set to: Release
CMake system: Linux
-- Target architecture: ARM64
-- Use NEON SIMD instructions
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Checking for module 'libgps'
--   Found libgps, version 3.22
-- Found GPSD: /usr/lib/aarch64-linux-gnu/libgps.so
-- Checking for module 'hamlib'
--   Found hamlib, version 4.3.1
CMake Warning (dev) at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:273 (message):
  The package name passed to `find_package_handle_standard_args` (HAMLIB)
  does not match the name of the calling package (hamlib).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  cmake/modules/Findhamlib.cmake:55 (find_package_handle_standard_args)
  CMakeLists.txt:260 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found HAMLIB: /usr/local/lib/libhamlib.so  
-- Found ALSA: /usr/lib/aarch64-linux-gnu/libasound.so (found version "1.2.4") 
-- Checking for module 'libudev'
--   Found libudev, version 247
CMake Warning (dev) at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:273 (message):
  The package name passed to `find_package_handle_standard_args` (UDEV) does
  not match the name of the calling package (udev).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  cmake/modules/Findudev.cmake:68 (find_package_handle_standard_args)
  CMakeLists.txt:274 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found UDEV: /usr/lib/aarch64-linux-gnu/libudev.so  
-- Configuring done
-- Generating done
-- Build files have been written to: /media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build

pi@buildpi:/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build $ grep -R archive.tar *
data/download_symbols-new-prefix/src/download_symbols-new-stamp/extract-download_symbols-new.cmake:get_filename_component(filename "/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar" ABSOLUTE)
data/download_symbols-new-prefix/src/download_symbols-new-stamp/download-download_symbols-new.cmake:       file='/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar'")
data/download_symbols-new-prefix/src/download_symbols-new-stamp/download-download_symbols-new.cmake:  file("" "/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar" actual_value)
data/download_symbols-new-prefix/src/download_symbols-new-stamp/download-download_symbols-new.cmake:    /media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar
data/download_symbols-new-prefix/src/download_symbols-new-stamp/download-download_symbols-new.cmake:if("/media/pi/500GB/usr/src/archive/Rpi-scratch/tmp/direwolf/build/data/archive.tar" STREQUAL "")
.
.
.

So the new version of cmake seems to be causing this problem.

-Steve

[dranch]

Yup.. I see it too. Very strange. To work around this for now, either don't run "make update-data" and this is technically NOT a requirement for a build or packaging. It just allows the older builds to have up to date understanding of APRS symbols, etc. If you've already tried to do the update, you can restore the original files included in the original git clone/checkout with:

cd build
mv data/symbols-new.txt.old data/symbols-new.txt
mv data/symbolsX.txt.old data/symbolsX.txt
mv data/tocalls.txt.old data/tocalls.txt

After that, direwolf v1.6 builds and packages just fine on Bullseye

[AG7GN]

Thanks, David.

-Steve