Required stack
필요 기술
JavaLinuxCloud ComputingSoftware ArchitectureAmazon Web ServicesApplication Performance MonitoringAutomationAPI DevelopmentAWS EC2Distributed SystemsConcurrency ControlAPI Error HandlingPerformance TuningMonitoring & LoggingTelegram IntegrationProxy ManagementWebSocket
Project brief
프로젝트 내용
1- Update application calls should have zero delay so that calls go at full speed.
2- Latency is very important — locally it's 250-300ms, so after deploying on AWS it should be minimum 200ms. So delay = 0 and latency should be low so calls hit fast.
3- We need to pick jobs, and for that we need to eliminate 429, 403, and 504 errors. Why does 429 come? Because we reuse the same token too fast. 403 comes when multiple calls hit from the same IP or combination calls hit together. 504 doesn't come that often, so for that we can keep a retry mechanism.
4- To eliminate 429, we need to use multiple tokens, because a job will only be picked when calls go without even a single 429, with latency 200-250ms and delay 0.
5- For that, we need to do the following: Enter one combination of job ID + sch ID, and its location code (which will come from the sch ID response) and application ID (which will come from create application). We'll add those two manually. Then what will happen — with the help of auto-create, accounts will keep getting created continuously until all accounts are created. Like if I enter 200, it will create 200 accounts.
6- What will the pipeline do? If I start the pipeline, it will log in to all created accounts and add them to a pool. Like if I enter 300, it will log in to 300 accounts. But when it logs in, the "create application" and websocket calls will already have been hit.
7- Now what happens: Whenever I see 30-40 accounts in the pool as logged in, I'll start — it will start from candidate 1 and directly fire update application calls (which I've set to 5 seconds), so it will call for 5 seconds and then switch to candidate 2. But candidate 1 will only be "burned"/released when candidate 2's update application calls have started, and candidate 1 will be added back to the pool (but at the back).
8- So candidates always switch like 2→3→4→5→6... if it reaches 200, it starts again from 1. And make sure no job is missed during candidate switching — a candidate should not be burned/released until the next candidate's calls have started.
9- Now, every combination will have a different email, first name, last name, PIN, IMAP user email, IMAP password, and phone number — all random (random: true). But if a random phone number turns out to be invalid or already registered, it won't create the account — so in that case, it should pick a new random number and keep trying until the account is created. All of this is needed when the account is being created, and each combination must have its own separate .config.json file.
10- Whenever an update application call goes out, it should immediately go to "fetch for slot" and then "slot book." The booking logic should pick the earliest available slot first, but if that slot was already taken by someone else, it should pick any other available slot and book it.
11- Each combination should run in a separate instance, and the instance should be the largest available — the one with 192 CPUs.
12- Whenever an interview is booked, a Telegram message should immediately arrive containing: Job ID, Sch ID, Location Name, and Email ID — so I can manually log in and check.
13- The same process should work for every combination — the same cycle.
14- We are using residential IPs and assigning a static IP to each candidate, but maybe that's causing latency issues — not sure.
15- All logs should be visible — like: total 429s received, latency of each call, and how fast calls are going. This should run continuously and never close until I manually stop it.
2- Latency is very important — locally it's 250-300ms, so after deploying on AWS it should be minimum 200ms. So delay = 0 and latency should be low so calls hit fast.
3- We need to pick jobs, and for that we need to eliminate 429, 403, and 504 errors. Why does 429 come? Because we reuse the same token too fast. 403 comes when multiple calls hit from the same IP or combination calls hit together. 504 doesn't come that often, so for that we can keep a retry mechanism.
4- To eliminate 429, we need to use multiple tokens, because a job will only be picked when calls go without even a single 429, with latency 200-250ms and delay 0.
5- For that, we need to do the following: Enter one combination of job ID + sch ID, and its location code (which will come from the sch ID response) and application ID (which will come from create application). We'll add those two manually. Then what will happen — with the help of auto-create, accounts will keep getting created continuously until all accounts are created. Like if I enter 200, it will create 200 accounts.
6- What will the pipeline do? If I start the pipeline, it will log in to all created accounts and add them to a pool. Like if I enter 300, it will log in to 300 accounts. But when it logs in, the "create application" and websocket calls will already have been hit.
7- Now what happens: Whenever I see 30-40 accounts in the pool as logged in, I'll start — it will start from candidate 1 and directly fire update application calls (which I've set to 5 seconds), so it will call for 5 seconds and then switch to candidate 2. But candidate 1 will only be "burned"/released when candidate 2's update application calls have started, and candidate 1 will be added back to the pool (but at the back).
8- So candidates always switch like 2→3→4→5→6... if it reaches 200, it starts again from 1. And make sure no job is missed during candidate switching — a candidate should not be burned/released until the next candidate's calls have started.
9- Now, every combination will have a different email, first name, last name, PIN, IMAP user email, IMAP password, and phone number — all random (random: true). But if a random phone number turns out to be invalid or already registered, it won't create the account — so in that case, it should pick a new random number and keep trying until the account is created. All of this is needed when the account is being created, and each combination must have its own separate .config.json file.
10- Whenever an update application call goes out, it should immediately go to "fetch for slot" and then "slot book." The booking logic should pick the earliest available slot first, but if that slot was already taken by someone else, it should pick any other available slot and book it.
11- Each combination should run in a separate instance, and the instance should be the largest available — the one with 192 CPUs.
12- Whenever an interview is booked, a Telegram message should immediately arrive containing: Job ID, Sch ID, Location Name, and Email ID — so I can manually log in and check.
13- The same process should work for every combination — the same cycle.
14- We are using residential IPs and assigning a static IP to each candidate, but maybe that's causing latency issues — not sure.
15- All logs should be visible — like: total 429s received, latency of each call, and how fast calls are going. This should run continuously and never close until I manually stop it.