Dense CRF with DGM

Semantic Image Segmentation with Conditional Random Fields
Zeinab Ghassabi
Posts: 12
Joined: Sun Dec 10, 2017, 04:31
Contact:

Dense CRF with DGM

Postby Zeinab Ghassabi » Sun Dec 10, 2017, 04:39

Good Day!
I employ DGM for medical image segmentation. Thanks in advance for providing such useful toolbox. Do you have any dense crf version for DGM?
I want to improve it so that it becomes a fully connected graphical model as Philip Krahenbuhl's paper. I wonder if you kindly please guide me in this point.

User avatar
Creator
Posts: 138
Joined: Tue Dec 16, 2008, 21:52
Location: Hannover, Germany
Contact:

Re: Dense CRF with DGM

Postby Creator » Sun Dec 17, 2017, 02:57

Thank you for the interest to the DGM library.

I have revised the dense-CRF Code, to make it work with OpenCV and DGM library. You can download it here: https://github.com/ereator/Dense-CRF

Currently I work on integrating the DGM library into the Dense-CRF. For that you Need to extract the potential Matrices. For example, now it is possible to save and read These Matrices in DGM as: http://research.project-10.de/dgmdoc/a0 ... 2cf6f688a2 and http://research.project-10.de/dgmdoc/a00618.html
I hope this Information will help you.

Zeinab Ghassabi
Posts: 12
Joined: Sun Dec 10, 2017, 04:31
Contact:

Re: Dense CRF with DGM

Postby Zeinab Ghassabi » Mon Dec 18, 2017, 01:47

So, if I include serialize.h in Demo train.cpp of DGM and use "to", I can save Potential matrices in that dat format. Further I can use it in the modified version of densecrf.
How long it takes to produce DGM with Denscrf? I wonder if you kindly please share with me the source code that I have to write in DGM to save the matrices.

User avatar
Creator
Posts: 138
Joined: Tue Dec 16, 2008, 21:52
Location: Hannover, Germany
Contact:

Re: Dense CRF with DGM

Postby Creator » Tue Dec 19, 2017, 18:02

So, if I include serialize.h in Demo train.cpp of DGM and use "to", I can save Potential matrices in that dat format. Further I can use it in the modified version of densecrf.

That is right.

How long it takes to produce DGM with Denscrf?

Do you mean how long will it take to implement DenseCRF into the DGM library? I plan to build it in version 1.6.0. With my current speed of the development it will be at the end of this year.
Or do you mean how long the calculation process takes? The calculation and saving the potentials in DGM takes about 1 – 10 seconds, depending on the image size. The DenseCRF inference takes about 1 – 5 seconds.

I wonder if you kindly please share with me the source code that I have to write in DGM to save the matrices.

It is only one string of code. For example in Demo Train:

Code: Select all

#include "DGM/serialize.h"
...
// ==================== STAGE 3: Filling the Graph =====================
Timer::start("Filling the Graph... ");
Mat nodePotentials = nodeTrainer->getNodePotentials(test_fv);
Serialize::to("fileName.dat", nodePotentials); // <<<< THIS LINE
graph->setNodes(nodePotentials);
graph->fillEdges(edgeTrainer, test_fv, params, params_len);
Timer::stop();

Zeinab Ghassabi
Posts: 12
Joined: Sun Dec 10, 2017, 04:31
Contact:

Re: Dense CRF with DGM

Postby Zeinab Ghassabi » Tue Dec 26, 2017, 14:44

Thank you for the response.

I downloaded two version of DGM:
  • The old one DGM v.1.5.1 which does not include serialize.h. In this program, I use a feature vector which is an array of 36 matrices for image classification. It works properly.
  • However, in the DGM-master (with opencv-3.2.0), which include serialize.h, when I use the above mentioned feature vector, the program does not work and has the following error:
    DGM_ASSERT_MSG(!isnan(node->Pot.at<float>(s, 0)), "The lower precision boundary for the potential of the node %zu is reached.\n

    in the MessagePassing.cpp

    I explore all the features to be sure whether they are between 0,255. My code works in DGM v.1.5.1 when I use all 36 features. However, when I just use each element of the array of 36 matrices, i.e. one matrix as a feature vector or some of them for classification, DGM-master works.

This is my code:

Code: Select all

// ----------- training image feature vector -----------
Mat FeatureSets;
vec_mat_t channels;

cv::Ptr<cv::ml::TrainData> raw_data = cv::ml::TrainData::loadFromCSV("E:\\zghassabi\\Project1-DIE\\Dataset\\image 1\\All_features.csv", 0);
cv::Mat data1 =raw_data->getSamples();
cv::Mat data2 = raw_data->getResponses();
cv::hconcat(data1, data2, data1);
cv::Mat data;
data1.convertTo(data, CV_8UC1);

int i = 0;
int colsize = 1024;
int rowsize = 61;

for (i =0; i < data.cols; i += colsize)  {
    cv::Mat m = data.colRange(i, i + colsize);
    channels.push_back(m);
    imshow("Image", m);
    cvWaitKey(0);
}

merge(channels, FeatureSets);
Mat train_fv = FeatureSets; resize(train_fv, train_fv, imgSize, 0, 0, INTER_LANCZOS4);

When I write

Code: Select all

for (i =0; i < 5120; i += colsize)
and

Code: Select all

nfeatures=5
it works.

Would you please inform me how to fix this error?

User avatar
Creator
Posts: 138
Joined: Tue Dec 16, 2008, 21:52
Location: Hannover, Germany
Contact:

Re: Dense CRF with DGM

Postby Creator » Sat Dec 30, 2017, 02:19

The error in MessagePassing.cpp is related to the inference stage. Having the current information, you shared with me, it is difficult to say, where exactly the problem lies. It might be in the node training phase, or data preparation phase. Could you please provide some additional information, in particular to the following questions:
  • Do you save and/or load the training data in-between the training and classification stages?
  • The error occurs on the first iteration or after some iteration?
  • Which inference method do you use? Have you tried running different inference methods?
  • Could you please check that the node potentials, used in the graph filling stage are not empty?
  • Which node and which edge training models do you use?
  • You wrote that you have checked the features. Have you also checked the responses in

    Code: Select all

    Mat data2 = raw_data->getResponses();
    In particular for the class, starting from which, the bug appears?

Ideally, I could help you more effectively, if I could reproduce the error. As I said it is difficult to define the source of this bug without debugging the code. (I assume, that since it works in DGM v.1.5.1 and does not work in DGM v.1.5.2, it is a bug).

Zeinab Ghassabi
Posts: 12
Joined: Sun Dec 10, 2017, 04:31
Contact:

Re: Dense CRF with DGM

Postby Zeinab Ghassabi » Sun Dec 31, 2017, 19:58

Thank you very much for your response.
regarding first question:
Do you save and/or load the training data in-between the training and classification stages?

Code: Select all

nodeTrainer->train();
nodeTrainer->save("E:\\zghassabi\\Project1-DIE\\Dataset\\image 1\\nodetrainn.dat");
nodeTrainer->reset();
nodeTrainer->load("E:\\zghassabi\\Project1-DIE\\Dataset\\image 1\\nodetrainn.dat");

edgeTrainer->train();
Timer::stop();

// ==================== STAGE 3: Filling the Graph =====================
Timer::start("Filling the Graph... ");
// CV_32FC(nStates) <- CV_8UC(nFeatures);
Mat nodePotentials = nodeTrainer->getNodePotentials(test_fv); // Classification: CV_32FC(nStates) <- CV_8UC(nFeatures)
Serialize::to("E:\\zghassabi\\Project1-DIE\\Dataset\\image 1\\fileName.dat", nodePotentials);
graph->setNodes(nodePotentials); // Filling-in the graph nodes
graph->fillEdges(edgeTrainer, test_fv, params, params_len); // Filling-in the graph edges with pairwise potentials
Timer::stop();


regarding second question:
The error occurs on the first iteration or after some iteration?

When I choose the below three sections in the node-training-model, the decoding part does not work properly. It seems that after several iteration the error appears.
section 1: Gaussian Mixture model
section 3: nearest neighbor
section 5: Microsoft random Forrest.

regarding third question:
Which inference method do you use? Have you tried running different inference methods?

I explored different combination of node taring models with edge training models. It seems that when the number of features is more than 5, GMM, k-nn, mRF do not work. I tested all of these combination with 1D feature . All of them work when the feature is 1D or 3D.

Could you please check that the node potentials, used in the graph filling stage are not empty?

nodePotentials is not empty.

Which node and which edge training models do you use?

node-training-model: OpenCV Gaussian Mixture Model
edge-training-model: Concatenated Model
the above models do not produce errors for 5-D feature.

You wrote that you have checked the features. Have you also checked the responses in

Code: Select all

Mat data2 = raw_data->getResponses();
In particular for the class, starting from which, the bug appears?

When I choose node training model as
section 1: Gaussian Mixture model
section 3: nearest neighbor
section 5: Microsoft random Forrest.
and when I use a feature matrix with higher dimensions.

Zeinab Ghassabi
Posts: 12
Joined: Sun Dec 10, 2017, 04:31
Contact:

Re: Dense CRF with DGM

Postby Zeinab Ghassabi » Mon Jan 01, 2018, 20:52

I attach also the files:
  1. input image, its ground truth, a text file of features for input image (for training)
  2. test-image, a text file of features for test image ( for testing)
the size of images and GT is 61 x 1024 (You can use grey-scale)
the size of features is 61 x 1024 x 36

I attached the modified source code of demo train, too.
Using these files, you can see the error.
sorry for inconvenience.
Attachments
All_features_testimage.csv
(7.43 MiB) Downloaded 35 times
All_features.csv
(6.59 MiB) Downloaded 38 times
dgm-demo train.txt
(13.33 KiB) Downloaded 38 times
mask_DABAB8D0.tif
mask_DABAB8D0.tif (37.14 KiB) Viewed 1185 times
DABAB8D0-gt4.tif
DABAB8D0-gt4.tif (1.63 KiB) Viewed 1185 times
DABAB8D0-Adjacentl.tif
DABAB8D0-Adjacentl.tif (61.74 KiB) Viewed 1185 times

User avatar
Creator
Posts: 138
Joined: Tue Dec 16, 2008, 21:52
Location: Hannover, Germany
Contact:

Re: Dense CRF with DGM

Postby Creator » Sat Jan 06, 2018, 21:52

I have tried out your code with 5 and 36 features and could not reproduce the bug. In both cases the code produces results. I have used the latest version of the DGM library from GitHub with OpenCV v.3.3.0.

However, I think that there might be 2 reasons for the bug:
  1. OpenCV v.3.2.0 had a bug with the data serialization:
    https://github.com/Project-10/DGM/issues/8,
    https://github.com/opencv/opencv/issues/9125.
    But this bug was confirmed only for Random Forest classifier. Maybe it is also relevant for the ml::TrainData::loadFromCSV() function.
    So I have disabled the nodeTrainer->save() / nodeTrainer->load() functionality.
  2. When using a lot of features, CTrainNodeCvGMM class may produce very small potentials. And the small potentials in its turn may lead to the “The lower precision boundary for the potential of the node %zu is reached.” problem. So, I would suggest to use another node training model, for example k-nearest neighbors: CTrainNodeCvKNN
Please see attached results:
CRF_output_CvGMM_f5.tif
Using CTrainNodeCvGMM and CTrainEdgeConcat and 5 features
CRF_output_CvGMM_f5.tif (87.77 KiB) Viewed 1162 times

CRF_output_CvGMM_f36.tif
Using CTrainNodeCvGMM and CTrainEdgeConcat and 36 features
CRF_output_CvGMM_f36.tif (79.73 KiB) Viewed 1162 times

CRF_output_CvKNN_f36.tif
Using CTrainNodeCvKNN and CTrainEdgePotts and 36 features
CRF_output_CvKNN_f36.tif (79.85 KiB) Viewed 1162 times

CRF_GT.tif
The visualized groundtruth
CRF_GT.tif (78.77 KiB) Viewed 1162 times


I attach also the code, that generates these results:
Demo Train.cpp
(9.31 KiB) Downloaded 38 times
I hope that will help you.

P.S.
I have noticed that the features data from the .csv files has 61 x 37887 samples and 61 x 1 responses. So after the concatenation you have 61 x (37*1024) data matrix, where the last 1024 values in every row are a mixture between samples and responses. Since, you use only 36 first features, it should not be a problem, however, that data container looked a bit strange to me.

Zeinab Ghassabi
Posts: 12
Joined: Sun Dec 10, 2017, 04:31
Contact:

Re: Dense CRF with DGM

Postby Zeinab Ghassabi » Sat Jan 13, 2018, 16:04

Thank you very very much for your kind and help!
I used OpenCV v.3.3.0 and it works properly on my computer, too. Thank you for improving the code.

I used CRF_output_CvKNN_f36.tif: Using CTrainNodeCvKNN and CTrainEdgePotts and 36 features. I also save the potential matrices in filename.dat to use in the densecrf. Would you please help me in modyfing your code for my images? I have nStates = 4. I can not understand the upper bound of m = 21 and i = 20. This is because you have 21 nStates? If I change nStates to 4, can I make comment this part:

Code: Select all

if (m == 17) continue;
      for (word i = 11; i <= 20; i++) {
         if (m == 6 && i == 17) continue;
         if (m == 18 && i == 9) continue;

why do you wrote the above part?

I attached you my code. The result is worse than DGM.

Thank you for your help.
Attachments
solution-DensCRF_output.tif
solution-DensCRF_output.tif (5.43 KiB) Viewed 1144 times
dense_inference.cpp
(8.06 KiB) Downloaded 38 times


Return to “Direct Graphical Models”

Who is online

Users browsing this forum: No registered users and 1 guest