Nov 27, 2017
deixa eu incluir seus beijos em minha rotina
seen from United States
seen from United Kingdom
seen from Australia
seen from Australia
seen from Australia
seen from Australia
seen from Morocco
seen from Australia
seen from Sri Lanka
seen from Honduras
seen from United States
seen from Colombia
seen from United Kingdom
seen from Malaysia
seen from United Kingdom
seen from Brazil
seen from Singapore
seen from Singapore
seen from Malaysia
seen from Türkiye
deixa eu incluir seus beijos em minha rotina
Semantic Image Processing for Developmental Data Science
NOTE: This is part of a four-part series. For more information, see Phase 1, Phase 2 and Phase 3.
Aim:
- I and my mentor Dr. Bradly Alicea worked towards segmenting the cells of C.Elegans during its embryogenesis process. This project is part of the DevoWorm group which acts as a sub organisation under the OpenWorm foundation. The DevoWorm group mainly focuses on the development process of a single cell to a complete organism.
- This project is an extension to the work done in GSoC 2017, where some image processing techniques were explored. This time the aim was to explore how machine learning/deep learning models can do this task.
Please refer to the original project idea and my accepted proposal.
What has been done:
1. Developed a plugin to ImageJ which does semantic image segmentation on SPIM images of C. Elegans.
-- The code for the plugin can be found here. The description of each of the file and how to use the plugin can be found here.
-- Please find and download the jar file for the plugin from here.
2. Built a web app (backend: Flask REST API, frontend: HTML/CSS, JavaScript, and AJAX) to calculate the centroids and area of each of the cells in each of the uploaded image and stores it in a downloadable CSV file. Even the processed images can be downloaded. Deployed the app on Heroku platform.
-- The code for the web app is pushed here.
-- The app is hosted here: https://open-devo-cell.herokuapp.com
What has to been done:
1. Try different architectures of deep learning models for unsupervised semantic image segmentation/semi-supervised semantic image segmentation to enhance the segmentation result.
2. The CSV file containing the cell positions and areas is generated. We can compare it with a reference CSV file which has information about the cell positions ( calculated as an average from different embryogenesis videos) along with the cell name. With this information, there is a possibility to automatically tag cells of a test input embryogenesis video.
Other:
- Complete Code Base.
- Weekly meetings over the GSoC period.
- I want to thank Google, INCF and the OpenWorm foundation for this opportunity. I really have to thank my mentor very much for always guiding me and giving me the freedom and time to think and come up with solutions. I really had a great time over the last three months and have learned a lot.
Phase 3 Coding
NOTE: This is part of a four-part series. For more information, see Phase 1, Phase 2 and Final Submission.
Phase 3 of the coding period for GSoC'19 began on July 31st and ended on Aug 26th, 2019.
The goal of Phase 3: is to build a web app to segment the cells of C. Elegans embryos and calculate the centroids and areas of each of the cell and store it in a downloadable CSV file. Used the basics of image processing techniques like extracting hue channel, applying blurring, morphology processing, and opencv contours.
What has been done:
- Built a Flask REST API as backend and built the front-end with HTML/CSS, JavaScript, and AJAX.
- Deployed the app on Heroku platform.
- Added documentation to code and wrote blog posts. Prepare the entire work for submission.
Issues faced:
- Initially, did not have a clear idea of what might be a good thing to do for Phase 3. Tried implementing the cell-star algorithm. Also, tried using the Matlab plugin, Cell profiler plugin, and as a python package. None of them worked.
- Web app behaved differently on my local system and in production. There was apparently a bug with the generation of CSV file.
- Some cells were not getting detected.
Solutions:
- Finally ended up getting good results with basic image processing techniques.
- I had to use global variables when serving the web app as only python 3.6.2 was available in Heroku. I was using python 3.5.2.
- Worked on different types of blurring (median, bilateral), morphological processings (opening, gradient, closing), and thresholding. Could reduce the error rate and recognize most of the cells in each of all test images.
Phase 2 Coding
NOTE: This is part of a four-part series. For more information, see Phase 1, Phase 3 and Final Submission.
Phase 2 of the coding period for GSoC'19 began on June 29th and ended on July 30th, 2019.
The goal of Phase 2: is to complete the Unsupervised plugin development for ImageJ tool. As the goal of both the Semi-Supervised approach and the Unsupervised approach aimed to do semantic segmentation on the SPIM images, and the unsupervised approach is yet to be completely implemented, we decided to focus on only the unsupervised approach.
What has been done:
- Went with the option to reimplement the whole model in pure tensorflow which would not have any problems with DeepLearning4J or Lambda layers.
- Trained the deep learning model on Kaggle, downloaded the model to my system and imported that model into boilerplate ImageJ plugin code with the help of DeepLearning4J support for importing tensorflow models.
- Tried the K-means algorithm on the result images of the
Issues faced:
- BatchNormalization layers have been given an alias name in the newer versions of tensorflow. DeepLearning4j did not support this.
- Had to work on reproducibility. Meaning, when I retrained the model, I would always get different results for the image segmentation.
- As the neural network is large, setting the parameters debugging the neural network every time was very hard.
- Was interrupted by academic activities (Exams and Campus placements) during this time.
Solutions:
- Was forced to downgrade tensorflow to 1.13.10.
- Set the seed values for numpy and tensorflow to achieve reproducibility. Please refer to this blog post for more information.
- I manually set the parameters at first and conducted different training on the model. Please refer to this blog post for more information.
- I had to put in more efforts and work for longer hours to catch up on the schedule.
Phase 1 Coding
NOTE: This is part of a four-part series. For more information, see Phase 2, Phase 3 and Final Submission.
Phase 1 of the coding period for GSoC'19 began on May 28th and ended on June 28th, 2019
Initial Planning: The goal of Phase 1: is to create an ImageJ plugin for segmenting the cells of C. Elegans during its embryogenesis process in an Unsupervised way. The plan was to employ a deep learning model for this task. As this plugin needs to be written in Java, it would not be optimal/easy to build the whole deep learning model in Java (because it has less amount of deep learning resources).
- We could train the deep learning model in Python using Tensorflow/Keras and save the model. We could then import that model into our plugin (basically any Java application) with the help of DeepLearnign4J’s flexibility for importing Keras models and Tensorflow models.
- During the Community Bonding period, some work has been put into trying out Jython programming, but it has been marked as not efficient for this use case.
- This Wnet paper served as an inspiration for doing semantic image segmentation in an Unsupervised manner.
- This is the dataset with which the deep learning model has to be trained. Most of the things went as planned except for a couple of things which will be discussed here.
What has been done:
- Built an autoencoder model for segmenting the cells. Trained the model several times with a different set of parameters, loss functions, optimizers and deep learning frameworks(Tf.keras and keras) on Kaggle.
- Was able to get some nice results from the model. You can check them out here. All of these images are outputs from different versions of this model tested on this test image.
- Constructed the java pipeline for importing saved models into a java application.
- Setup a basic ImageJ plugin to which the java pipeline (for importing deep learning models) has to be added.
Issues faced:
- The soft-cut-normalized loss described in the Wnet paper was hard to implement. Even after implementing it, it did not seem to be very useful for the kind of data that is being fed to the model.
- Most of the documents and tutorial examples for deeplearnign4j are outdated.
- Had to buy some additional RAM for my laptop.
- Limitation from the DeepLearnign4J side. A custom convolutional layer has been added as the last layer of the model. Currently, it is not possible to import deep learning models which contain convolutional custom layers into Java applications (ImageJ for this project) with the help of DeepLearnign4J. They only support simple lambda computations like this.
Solutions:
- Since soft-cut-normalized loss did not give any better results, the last layer's activation has been changed from `softmax` to `relu`. The output images that you see are a result of this action.
- There is a community for deeplearning4j where the maintainers of DeepLearning4J help with issues regarding DeepLearning4J. They have helped navigate to the latest source code and examples for DeepLearning4J.
- Bought 4GB additional RAM. Now my laptop has 8GB.
- This is the main issue now and currently working on this. There are three options to go ahead.
- - 1st option is to try to convert the existing saved Keras model to a tensorflow model and see if it works.
- - 2nd option is to go back to the implementation of the autoencoder and remove the custom layer (which is not compatible with dl4j, but also is the heart of the model) and reproduce the same results by adding an inbuilt keras convolutional layer.
- - 3rd option is to implement the whole model in pure tensorflow and see how good the results will be. I will work on these three options this week and will update the progress.
Coding Period (Week - 1)
May 25 - May 31
This marks the beginning of the coding period.
What Have I done:
1. Implemented the W-net deep learning model architecture. Started preparing the dataset by using image augmentation techniques. Worked on writing custom loss function (soft cut normalized loss) for training the encoder and reconstruction loss for training the whole autoencoder.
2. Discussed the ideas for phase 3 of the GSoC phase. We have considered the four ideas
2.1 Extending GSoC 2018 project to Geppetto. Requires a lot of engineering effort. So this is not recommended.
2.2 Create a visualization skin for GSoC 2018 project. Focus on networks and 3-D modeling. Significant time overhead for setting up the environment and learning about the technologies
2.3 Work with Morphozoic and CompuCell3D tool to make a visualization of an embryo of C. Elegans. Required Cpp experience and the work is more related to the visualization field.
2.4 Cell tracking and tagging from Brightfield embryogenesis movies. We have a video like this. We have to segment and track the cells as the process moves on. This idea is related to the computer vision field and this is something I might end up working on for phase 3 of GSoC.
3. Presented my research work during the community bonding period and explained my approach for the project to the OpenWorm community. You can find the presentation here.
What will I do next:
1. Continue working on the W-net paper implementation. Implement the post-processing techniques “Conditional Random Field smoothening” and “Hierarchical Segmentation”.
2. Train the W-net model on the available SPIM images and test the results.
Community Bonding (Week 2)
May 17 - May 24
Finally, I’m done with all my semester exams, academic projects, and reviews this week. That gave me a lot of relief. This post is to summarize all the work that I have done this week and my plans for next week.
What Have I done:
1. I started with Jython programming to see if I can do Image Processing with Python code right from the plugin. Basically, pass the image from the plugin code(Java) to Python, do the required processing and then return the processed image to the plugin for the user to see the results.
I wanted Python because I could build deep learning models easily with the help of the vast Deep Learning framework ecosystem available for Python. However, that did not go well as I found out that Jython is a bit ineffective in this case. See this thread.
Some other potential options are Weka and DeepLearning4J. Weka is more of a plug and play tool. It is useful for beginners to quickly try out basic ML/DL models. If one wants to customize the model and play with the parameters, it is not recommended. Refer this to learn more about the pros and cons of Weka.
I have tried out DeepLearning4J and it looks awesome. I implemented a basic CNN in Java and the training speed is exceptional. The only problem I faced is that the tutorials on their channel are outdated. I plan to stick with DeepLearning4J for this project.
2. I’m supposed to give a slide show presentation to my mentor on May 31st. I have prepared content for that.
3. Participated in the weekly hackathon conducted 17th May.
What will I do next:
1. Derive a solid objective for building the feature analysis tool. I’m a bit unclear about what would be a good thing to do for feature analysis (during phase 3 of GSoC).
2. Present the project. Report my progress and explain my approach for the project.
Cheers!!
Community Bonding (Week 1)
So, the results of GSoC’19 came out on May 6th, 2019 and I’m glad that I have been selected as a GSoC student with INCF.
To know a bit about my project, please look at my previous post.
What Have I done:
1. Have gone through the GSoC reading materials given by my mentor.
2. Setup the environment for developing plugins for ImageJ.
3. Downloaded this example plugin, made changes to the code, built the project, packaged it as a JAR file and finally added it as a plugin to ImageJ. I got myself familiar with the workflow.
4. Participated in two weekly hackathons (10th and 15th May). Have given an overview of my project to the OpenWorm community.
What will I do next:
1. Make use of ImageJ’s capability to add python scripts to plugins. Basically, try to manipulate an image with python code placed in the plugin.
2. See if I can add saved Deep Learning Models to ImageJ plugins with the help of Deeplearning4J.
3. Discuss the progress with my mentor and decide on what to do next.