Caffe: CMake source compilation (OpenCV4.2+CUDA)

I spent two days at home on Saturdays and Sundays, getting a glimpse of some of pycaffe's interfaces and api settings, but there's still a lot of detail to pay attention to for different layers, so I'm also going to write a blog that specifically summarizes pycaffe's common layer parameter settings. (LZ is silly, handwritten prototxt before, a bitter tear, silly/(o)/~)

I've written before about installing Caffe directly with Anaconda, so it's not good to install caffe. Why do you have to install the source code again?

1. The back end of the project is the tensorflow interface, the front end is the caffe interface, some parameter settings of Caffe or source code query is more convenient
2. Some layers are special, and this customized layer may need to be written in c before source code is compiled

However, there are many pits in source code compilation, and listen to LZ one by one:

Download the source code first. https://github.com/weiliu89/caffe

Install dependencies according to tutorial: http://caffe.berkeleyvision.org/installation.html

LZ's small notebook is Ubuntu 16.04

  1. So you need to install the corresponding dependent libraries first
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev

  1. LZ then uses CMake compiled, It requires CMake version >= 2.8.7. The basic steps are as follows:
mkdir build
cd build
cmake ..
make all
make install
make runtest

According to the tutorial, it looks like it's easy to do in 10 minutes, but that's not the case.

  1. Of course you have to set up CUDA
# ---[ Options
caffe_option(CPU_ONLY  "Build Caffe without CUDA support" OFF) # TODO: rename to USE_CUDA
caffe_option(USE_CUDNN "Build Caffe with cuDNN library support" ON IF NOT CPU_ONLY)
caffe_option(USE_NCCL "Build Caffe with NCCL library support" OFF)
caffe_option(BUILD_SHARED_LIBS "Build shared libraries" ON)
caffe_option(BUILD_python "Build Python wrapper" ON)
set(python_version "2" CACHE STRING "Specify which Python version to use")
caffe_option(BUILD_matlab "Build Matlab wrapper" OFF IF UNIX OR APPLE)
caffe_option(BUILD_docs   "Build documentation" ON IF UNIX OR APPLE)
caffe_option(BUILD_python_layer "Build the Caffe Python layer" ON)
caffe_option(USE_OPENCV "Build with OpenCV support" ON)
caffe_option(USE_LEVELDB "Build with levelDB" ON)
caffe_option(USE_LMDB "Build with lmdb" ON)
caffe_option(ALLOW_LMDB_NOLOCK "Allow MDB_NOLOCK when reading LMDB files (only if necessary)" OFF)
caffe_option(USE_OPENMP "Link with OpenMP (when your BLAS wants OpenMP and you get linker errors)" OFF)
  1. First operate normally:
cmake ..

Notice the high energy ahead (ERROR coming)

  1. The first pit is here. In order to compile tensorflow source code before LZ, protobuf has been installed in the source code, which also uses the standard -std=c++11.
    So some errors like the following were reported in cmake
error: #error This file requires compiler and library support for the ISO C++ 2011 standard. 
This support is currently experimental, and must be enabled with the -std=c++11 or
 -std=gnu++11 compiler options
Error:'nullptr'was not declared in this scope

So the first step is to add -std=c++11, cmakel

Makefile has changed this way, but LZ in makefile hasn't tried it. If a little partner has tried it, welcome guidance

CXXFLAGS += -pthread -fPIC $(COMMON_FLAGS) $(WARNINGS) -std=c++11
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS) -std=c++11
LINKFLAGS += -pthread -fPIC $(COMMON_FLAGS) $(WARNINGS) -std=c++11
  1. Second possible error:
nvcc fatal : Unsupported gpu architecture 'compute_20'

This is determined by the CUDA version, LZ version 10.1, this file is in makefile.config, by the way

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
                -gencode arch=compute_20,code=sm_21 \
                -gencode arch=compute_30,code=sm_30 \
                -gencode arch=compute_35,code=sm_35 \
                -gencode arch=compute_50,code=sm_50 \
                -gencode arch=compute_52,code=sm_52 \
                -gencode arch=compute_60,code=sm_60 \
                -gencode arch=compute_61,code=sm_61 \
                -gencode arch=compute_61,code=compute_61
//Comment out these two sentences:
 -gencode arch=compute_20,code=sm_20 \
 -gencode arch=compute_20,code=sm_21 \
  1. Third possible error:
fatal error: hdf5.h: No such file or directory compilation terminated.

Use locate to locate your hdf5.h location and add the corresponding path in makefile

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/local/include /usr/include/hdf5/serial/
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib  /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

Modify library

LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial
  1. Fourth possible error:

This problem is caused by the interface change in OpenCV version 4, which is a little bit more and needs to be replaced with all caffe sources related to this CV_LOAD_IMAGE_COLOR.

../caffe/src/caffe/layers/window_data_layer.cpp:293:42: error: 'CV_LOAD_IMAGE_COLOR' was not declared in this scope
         cv_img = cv::imread(image.first, CV_LOAD_IMAGE_COLOR);

src/caffe/layers/window_data_layer.cpp

          image_database_cache_[window[WindowDataLayer<Dtype>::IMAGE_INDEX]];
        cv_img = DecodeDatumToCVMat(image_cached.second, true);
      } else {
        //cv_img = cv::imread(image.first, CV_LOAD_IMAGE_COLOR);
        cv_img = cv::imread(image.first, cv::IMREAD_COLOR);
        if (!cv_img.data) {
          LOG(ERROR) << "Could not open or find file " << image.first;
          return;

src/caffe/test/test_io.cpp

bool ReadImageToDatumReference(const string& filename, const int label,
    const int height, const int width, const bool is_color, Datum* datum) {
  cv::Mat cv_img;
  //int cv_read_flag = (is_color ? CV_LOAD_IMAGE_COLOR :
    //CV_LOAD_IMAGE_GRAYSCALE);
  int cv_read_flag = (is_color ? cv::IMREAD_COLOR :
    cv::IMREAD_GRAYSCALE);

  cv::Mat cv_img_origin = cv::imread(filename, cv_read_flag);
  if (!cv_img_origin.data) {

src/caffe/util/io.cpp

void WriteProtoToBinaryFile(const Message& proto, const char* filename) {
cv::Mat ReadImageToCVMat(const string& filename,
   const int height, const int width, const bool is_color) {
 cv::Mat cv_img;
// int cv_read_flag = (is_color ? CV_LOAD_IMAGE_COLOR :
  // CV_LOAD_IMAGE_GRAYSCALE);
 int cv_read_flag = (is_color ? cv::IMREAD_COLOR :
   cv::IMREAD_GRAYSCALE);
 cv::Mat cv_img_origin = cv::imread(filename, cv_read_flag);
 if (!cv_img_origin.data) {
   LOG(ERROR) << "Could not open or find file " << filename;


cv::Mat DecodeDatumToCVMat(const Datum& datum, bool is_color) {
 CHECK(datum.encoded()) << "Datum not encoded";
 const string& data = datum.data();
 std::vector<char> vec_data(data.c_str(), data.c_str() + data.size());
// int cv_read_flag = (is_color ? CV_LOAD_IMAGE_COLOR :
  // CV_LOAD_IMAGE_GRAYSCALE);
 int cv_read_flag = (is_color ? cv::IMREAD_COLOR :
   cv::IMREAD_GRAYSCALE);
 cv_img = cv::imdecode(vec_data, cv_read_flag);
 if (!cv_img.data) {
   LOG(ERROR) << "Could not decode datum ";
  1. The fifth error you may encounter:
    Anaconda's three-party library conflicts with the local library, so comment out Anaconda's environment variables, compile them, and then release them.

Finally, the successfully compiled diagram is shown.

I trampled thousands of pits myself, jumped out of one and entered another!

It is said that there were security risks before OpenCV4.1, so now LZ companies are switching to OpenCV4.2.

Reference address:
1.https://blog.csdn.net/m_buddy/article/details/88085425
2.https://blog.csdn.net/qq_18649781/article/details/89047437
3.https://github.com/BVLC/caffe/issues/6680
4.https://github.com/BVLC/caffe/pull/6625/files/7f503bd9a19758a173064e299ab9d4cac65ed60f

PS:
LZ has been quarantined in his own home for more than 20 days, and has to be quarantined for 14 days when he gets to work. There are no two doors left for sale. I wish I could press the road O() in a sunny afternoon without fear.

It's a critical time. You can't relax. The more important it is, the more you can hold it up!

Go to Wuhan!HuBei refueling!Go China!_(_)_

307 original articles were published, 208 were praised, 600,000 visits+
His message board follow

Tags: cmake Anaconda github Python

Posted on Sun, 23 Feb 2020 16:15:05 -0800 by Juan Dela Cruz