Background: Carcinoembryonic antigen is overexpressed in colorectal cancer (CRC), making it an optimal target for fluorescence imaging. A phase I/II study was designed to determine the optimal imaging dose of SGM-101 for intraoperative fluorescence imaging of primary and recurrent CRC. Methods: Patients were included and received a single dose of SGM-101 at least 24 h before surgery. Patients who received routine anticancer therapy (i.e., radiotherapy or chemotherapy) also were eligible. A dedicated near-infrared imaging system was used for real-time fluorescence imaging during surgery. Safety assessments were performed and SGM-101 efficacy was evaluated per dose level to determine the most optimal imaging dose. Results: Thirty-seven patients with CRC were included in the analysis. Fluorescence was visible in all primary and recurrent tumors. In seven patients, no fluorescence was seen; all were confirmed as pathological complete responses after neoadjuvant therapy. Two tumors showed false-positive fluorescence. In the 37 patients, a total of 97 lesions were excised. The highest mean intraoperative tumor-to-background ratio (TBR) of 1.9 (p = 0.019) was seen in the 10-mg dose. This dose showed a sensitivity of 96%, specificity of 63%, and negative predictive value of 94%. Nine patients (24%) had a surgical plan alteration based on fluorescence, with additional malignant lesions detected in six patients. Conclusions: The optimal imaging dose was established at 10 mg 4 days before surgery. The results accentuate the potential of SGM-101 and designated a promising base for the multinational phase III study, which enrolled the first patients in June 2019.