Tonmoay Deb

I am a second-year Ph.D. student in Computer Science at Northwestern University. I am a member of Northwestern Security & AI Lab (NSAIL) and advised by Dr. V.S. Subrahmanian. I am exploring solutions on emerging defensive security by designing/applying algorithms in Artificial Intelligence, Machine Learning, Computer Vision, and Natural Language Processing.

Before joining Northwestern, I was a Master's (thesis-based) student in Computer Science at the University of New Hampshire. I spent Summer 2021 at the Center for Coastal and Ocean Mapping/NOAA-UNH Joint Hydrographic Center, working on Unsupervised Semantic Segmentation under Dr. Yuri Rzhanov and Dr. Kim Lowell.

Before that, I was a Research Associate (full-time) at the Artificial Intelligence and Cybernetics (AGenCy) Lab, Independent University, Bangladesh. I was supervised by Dr. Amin Ahsan Ali, Dr. A K M Mahbubur Rahman, and Dr. Iftekhar Tanveer. Some notable projects are Diversity in English Video Captioning, Video Captioning in the Bengali Language, and Scalable Bengali Speech-to-Text.

My research interest evolved since my Undergrad at North South University. One of my key projects was PATRON. I was blessed with excellent advisors, Dr. Mohammad Rashedur Rahman, Mr. Adnan Firoze, and Dr. Mohammad Ashrafuzzaman Khan.

I dream to give dreams to machines, to solve the real-world problems!

Email  /  Google Scholar  /  GitHub  /  LinkedIn  /  Old Site

• June 21, 2023: Started Summer Research Internship (Computer Vision) @ Zillow Inc.!


• December 21, 2022: Received AAAI Conference Travel Grant!


• December 10, 2022: Received MS in Computer Science at Northwestern!


• December 02, 2022: I am on the Northwestern McCormick School of Engineering News! Link


• October 25, 2022: DUCK Demo paper accepted at AAAI 2023!


• October 20, 2022: Presented DUCK at Conference on AI & National Security Link, Tweet


• February 02, 2022: Got married on 2/2/2022!


• October 04, 2021: One paper accepted at WACV 2022!


• September 20, 2021: Joined Northwestern for Ph.D. in Computer Science


• May 18, 2021: Started Summer Research Internship at CCOM, University of New Hampshire


• April 24, 2021: One paper from PATRON accepted at IJCNN 2021!


• February 1, 2021: Starting Masters in Computer Science (thesis-based) at University of New Hampshire

We have developed the DUCK multi-agent testbed that security agencies can use to simulate drone-based attacks by diverse actors and develop a combination of surveillance camera, drone, and cyber defenses against them.

We proposed a novel approach for diverse video captioning followed by extensive analysis and comparison with recent works.

[Full Text pdf] / [Presentaton Slides] / [Video Demo]

We proposed a novel crowd group tracking benchmark dataset + algorithm.

The main goal was to reduce the effort of coral image annotation as it is a very expensive (requires domain experts) and time-consuming process. Also, one major limitation was that the existing annotations are sparse (200 pixels per image). Another major challenge was that the coral objects don't conform to any specific shape, so the regular classifier can not locate even with sufficient samples. I first tried to overcome the sample shortage issue by integrating few-shot learning for recognizing class labels because patch-based methods are erroneous due to having fixed patch sizes. After satisfactory output, I moved to Unsupervised Segmentation of the corals because we had to distinguish the coral objects precisely. However, due to high variance among samples, I later implemented weakly-supervised semantic segmentation, e.g., 10-pixel labels per training, which performed relatively better. The work is ongoing.

The template is taken from Jon Barron