In a Market Where Being 3 Seconds Faster Wins

The first challenge was each site's security barriers. Musinsa, Nike, Kasina, Adidas... hundreds of sites each block bots in different ways. Some use Cloudflare, some use proprietary captcha, some implement IP rate limiting. Breaking through required deep understanding of the N/W layer. We analyzed each site's authentication flow, session management, and request patterns to design optimized bypass strategies per site.

But we couldn't use browser engines like Selenium. They're accurate but too slow. Launching a real browser, rendering pages, parsing DOM - precious seconds wasted in each step. So we developed direct HTTP requests with precisely matched headers, cookies, and auth tokens for each site. We built our own lightweight crawler engine with proxy manager, header generator, and retry logic.

The key was the 'Template Method Pattern'. The framework handles common crawling logic (proxy management, header generation, error handling, retries), while site-specific parts (parsing logic, auth logic) are separated into modules. This means when a new site is added or an existing site changes structure, only that module needs modification. No need to touch the entire system. And any developer can modify 'the collection part, the parsing part' with minimal learning curve and apply it to the system.

We solved the message loss problem with Kafka. Regular webhook methods lose messages when there's network instability or receiver-side failures. By implementing a Kafka-based message queue, we created a structure that delivers as fast as possible while retaining messages until receipt is confirmed. Result: 0% message loss rate.

The external notification system was designed with the concept of 'external channels'. Discord is just one channel. Slack, Telegram, webhook, email - any channel can be added as a plugin. When a customer wants a new notification channel, we just add the channel module without touching the existing system.

And we made all of this controllable from a web service. Which sites to monitor, which keywords to watch, where to send notifications - all configurable on the web without developers.

Real-time collection status is viewable on the logging monitor screen. When each crawler ran, how many products it collected, whether there were errors - all displayed in real-time. Problems are immediately visible, and individual crawlers can be toggled on/off directly from the web.

Collected data can be utilized like a search engine. Search indexed products by keyword, check price change history, filter products by specific conditions. Not just 'a system that sends notifications', but an extensible structure for 'a platform that accumulates and analyzes data'.